The MPack should function in a kerberized cluster (justinleet) closes 
apache/incubator-metron#518


Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/41b5b105
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/41b5b105
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/41b5b105

Branch: refs/heads/Metron_0.4.0
Commit: 41b5b1050cf29e5dcb5d0c36b5a9dbd1cafa745e
Parents: 55062fb
Author: justinjleet <justinjl...@gmail.com>
Authored: Mon Apr 17 15:16:49 2017 -0400
Committer: leet <l...@apache.org>
Committed: Thu Apr 20 16:03:29 2017 -0400

----------------------------------------------------------------------
 metron-deployment/Kerberos-manual-setup.md      | 217 +++++++++++++++++++
 metron-deployment/README.md                     |  15 ++
 .../common-services/METRON/CURRENT/metainfo.xml |   4 +-
 .../package/scripts/enrichment_commands.py      |  11 +
 .../CURRENT/package/scripts/metron_client.py    |   2 +-
 .../CURRENT/package/scripts/metron_service.py   |  16 --
 .../CURRENT/package/scripts/parser_commands.py  |   9 +-
 .../package/templates/client_jaas.conf.j2       |   2 +-
 .../CURRENT/package/templates/storm.config.j2   |   2 +-
 .../ambari-storm-site-properties.png            | Bin 0 -> 49605 bytes
 .../readme-images/ambari-storm-site.png         | Bin 0 -> 134251 bytes
 .../readme-images/custom-storm-site-final.png   | Bin 0 -> 128117 bytes
 .../enable-kerberos-configure-kerberos.png      | Bin 0 -> 136196 bytes
 .../readme-images/enable-kerberos-started.png   | Bin 0 -> 149382 bytes
 .../readme-images/enable-kerberos.png           | Bin 0 -> 16497 bytes
 metron-deployment/vagrant/Kerberos-setup.md     | 200 ++---------------
 .../ambari-storm-site-properties.png            | Bin 49605 -> 0 bytes
 .../vagrant/readme-images/ambari-storm-site.png | Bin 134251 -> 0 bytes
 .../readme-images/custom-storm-site-final.png   | Bin 128117 -> 0 bytes
 .../enable-kerberos-configure-kerberos.png      | Bin 136196 -> 0 bytes
 .../readme-images/enable-kerberos-started.png   | Bin 149382 -> 0 bytes
 .../vagrant/readme-images/enable-kerberos.png   | Bin 16497 -> 0 bytes
 22 files changed, 269 insertions(+), 209 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/Kerberos-manual-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-manual-setup.md 
b/metron-deployment/Kerberos-manual-setup.md
new file mode 100644
index 0000000..4eaa725
--- /dev/null
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -0,0 +1,217 @@
+# Setting Up Kerberos outside of an Ambari Management Pack
+The Ambari Management pack will manage Kerberization when used.
+**Note:** These are instructions for Kerberizing Metron Storm topologies from 
Kafka to Kafka. This does not cover the sensor connections or MAAS.
+General Kerberization notes can be found in the metron-deployment 
[README.md](README.md)
+
+## Setup the KDC
+See [Setup the KDC](vagrant/Kerberos-setup.md)
+
+4. Setup the admin and metron user principals. You'll kinit as the metron user 
when running topologies. Make sure to remember the passwords.
+    ```
+    kadmin.local -q "addprinc admin/admin"
+    kadmin.local -q "addprinc metron"
+    ```
+
+## Kerberize Metron
+
+1. Stop all topologies - we will  restart them again once Kerberos has been 
enabled.
+    ```
+    for topology in bro snort enrichment indexing; do storm kill $topology; 
done
+    ```
+
+2. Create the metron user HDFS home directory
+    ```
+    sudo -u hdfs hdfs dfs -mkdir /user/metron && \
+    sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
+    sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+    ```
+
+3. In [Ambari](http://node1:8080), setup Storm to run with Kerberos and run 
worker jobs as the submitting user:
+
+    a. Add the following properties to custom storm-site:
+    ```
+    
topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
+    
nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
+    supervisor.run.worker.as.user=true
+    ```
+
+    b. In the Storm config section in Ambari, choose “Add Property” under 
custom storm-site:
+
+    ![custom storm-site](readme-images/ambari-storm-site.png)
+
+    c. In the dialog window, choose the “bulk property add mode” toggle 
button and add the below values:
+
+    ![custom storm-site 
properties](readme-images/ambari-storm-site-properties.png)
+
+4. Kerberize the cluster via Ambari. More detailed documentation can be found 
[here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+
+    a. For this exercise, choose existing MIT KDC (this is what we setup and 
installed in the previous steps.)
+
+    ![enable keberos](readme-images/enable-kerberos.png)
+
+    ![enable keberos get started](readme-images/enable-kerberos-started.png)
+
+    b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal 
will end up as admin/ad...@example.com when testing the KDC. Use the password 
you entered during the step for adding the admin principal.
+
+    ![enable keberos 
configure](readme-images/enable-kerberos-configure-kerberos.png)
+
+    c. Click through to “Start and Test Services.” Let the cluster spin 
up, but don't worry about starting up Metron via Ambari - we're going to run 
the parsers manually against the rest of the Hadoop cluster Kerberized. The 
wizard will fail at starting Metron, but this is OK. Click “continue.” When 
you’re finished, the custom storm-site should look similar to the following:
+
+    ![enable keberos configure](readme-images/custom-storm-site-final.png)
+
+5. Setup Metron keytab
+    ```
+    kadmin.local -q "ktadd -k metron.headless.keytab met...@example.com" && \
+    cp metron.headless.keytab /etc/security/keytabs && \
+    chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+    chmod 440 /etc/security/keytabs/metron.headless.keytab
+    ```
+
+6. Kinit with the metron user
+    ```
+    kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
+    ```
+
+7. First create any additional Kafka topics you will need. We need to create 
the topics before adding the required ACLs. The current full dev installation 
will deploy bro, snort, enrichments, and indexing only. e.g.
+    ```
+    ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 
--create --topic yaf --partitions 1 --replication-factor 1
+    ```
+
+8. Setup Kafka ACLs for the topics
+    ```
+    export KERB_USER=metron
+    for topic in bro enrichments indexing snort; do
+        ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--topic ${topic}
+    done
+    ```
+
+9. Setup Kafka ACLs for the consumer groups
+    ```
+    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--group bro_parser
+    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--group snort_parser
+    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--group yaf_parser
+    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--group enrichments
+    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--group indexing
+    ```
+
+10. Add metron user to the Kafka cluster ACL
+    ```
+    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--cluster kafka-cluster
+    ```
+
+11. We also need to grant permissions to the HBase tables. Kinit as the hbase 
user and add ACLs for metron.
+    ```
+    kinit -kt /etc/security/keytabs/hbase.headless.keytab 
hbase-metron_clus...@example.com
+    echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+    echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+    ```
+
+12. Create a “.storm” directory in the metron user’s home directory and 
switch to that directory.
+    ```
+    su metron
+    mkdir ~/.storm
+    cd ~/.storm
+    ```
+
+13. Create a custom client jaas file. This should look identical to the Storm 
client jaas file located in /etc/storm/conf/client_jaas.conf except for the 
addition of a Client stanza. The Client stanza is used for Zookeeper. All 
quotes and semicolons are necessary.
+    ```
+    cat << EOF > client_jaas.conf
+    StormClient {
+        com.sun.security.auth.module.Krb5LoginModule required
+        useTicketCache=true
+        renewTicket=true
+        serviceName="nimbus";
+    };
+    Client {
+        com.sun.security.auth.module.Krb5LoginModule required
+        useKeyTab=true
+        keyTab="/etc/security/keytabs/metron.headless.keytab"
+        storeKey=true
+        useTicketCache=false
+        serviceName="zookeeper"
+        principal="met...@example.com";
+    };
+    KafkaClient {
+        com.sun.security.auth.module.Krb5LoginModule required
+        useKeyTab=true
+        keyTab="/etc/security/keytabs/metron.headless.keytab"
+        storeKey=true
+        useTicketCache=false
+        serviceName="kafka"
+        principal="met...@example.com";
+    };
+    EOF
+    ```
+
+14. Create a storm.yaml with jaas file info. Set the array of nimbus hosts 
accordingly.
+    ```
+    cat << EOF > storm.yaml
+    nimbus.seeds : ['node1']
+    java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
+    storm.thrift.transport : 
'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+    EOF
+    ```
+
+15. Create an auxiliary storm configuration json file in the metron user’s 
home directory. Note the login config option in the file points to our custom 
client_jaas.conf.
+    ```
+    cat << EOF > ~/storm-config.json
+    {
+        "topology.worker.childopts" : 
"-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+    }
+    EOF
+    ```
+
+16. Setup enrichment and indexing.
+
+    a. Modify enrichment.properties as root located at 
`${METRON_HOME}/config/enrichment.properties`
+    ```
+    if [[ $EUID -ne 0 ]]; then
+        echo -e "\nERROR:\tYou must be root to run these commands.  You may 
need to type exit."
+    else
+        sed -i 
's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' 
${METRON_HOME}/config/enrichment.properties
+        sed -i 
's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/'
 ${METRON_HOME}/config/enrichment.properties
+    fi
+    ```
+
+    b. Modify elasticsearch.properties as root located at 
`${METRON_HOME}/config/elasticsearch.properties`
+    ```
+    if [[ $EUID -ne 0 ]]; then
+        echo -e "\nERROR:\tYou must be root to run these commands.  You may 
need to type exit."
+    else
+        sed -i 
's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' 
${METRON_HOME}/config/elasticsearch.properties
+        sed -i 
's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/'
 ${METRON_HOME}/config/elasticsearch.properties
+    fi
+    ```
+
+17. Distribute the custom jaas file and the keytab to each supervisor node, in 
the same locations as above. This ensures that the worker nodes can 
authenticate.  For a one node cluster, nothing needs to be done.
+
+18. Kinit with the metron user again
+    ```
+    su metron
+    cd
+    kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
+    ```
+
+19. Restart the parser topologies. Be sure to pass in the new parameter, 
“-ksp” or “--kafka_security_protocol.” Run this from the metron home 
directory.
+    ```
+    for parser in bro snort; do
+        ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s 
${parser} -ksp SASL_PLAINTEXT -e storm-config.json
+    done
+    ```
+
+20. Now restart the enrichment and indexing topologies.
+    ```
+    ${METRON_HOME}/bin/start_enrichment_topology.sh
+    ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+    ```
+
+Metron should be ready to receieve data.
+
+## Push Data
+See [Push Data](vagrant/Kerberos-setup.md)
+
+### Other useful commands
+See [Other useful commands](vagrant/Kerberos-setup.md)
+
+#### References
+* 
[https://github.com/apache/storm/blob/master/SECURITY.md](https://github.com/apache/storm/blob/master/SECURITY.md)

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index 66aa5f4..6f1653b 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -146,6 +146,21 @@ Build the Ambari Mpack to get the dashboard updated 
appropriately.
 
 Once the MPack is installed, run the Kibana service's action "Load Template" 
to install dashboards.  This will completely overwrite the .kibana in 
Elasticsearch, so use with caution.
 
+## Kerberos
+The MPack can allow Metron to be installed and then Kerberized, or installed 
on top of an already Kerberized cluster.  This is done through Ambari's 
standard Kerberization setup.
+
+### Caveats
+* For nodes using a Metron client and a local repo, the repo must exist on all 
nodes (e.g via createrepo). This repo can be empty; only the main Metron 
services need the RPMs.
+* A Metron client must be installed on each supervisor node in a secured 
cluster.  This is to ensure that the Metron keytab and client_jaas.conf get 
distributed in order to allow reading and writing from Kafka.
+  * When Metron is already installed on the cluster, this should be done 
before Kerberizing.
+  * When addding Metron to an already Kerberized cluster, ensure that all 
supervisor nodes receive a Metron client.
+* Storm (and Metron) must be restarted after Metron is installed on an already 
Kerberized cluster.  Several Storm configs get updated, and Metron will be 
unable to write to Kafka without a restart.
+  * Kerberizing a cluster with an existing Metron already has restarts of all 
services during Kerberization, so it's unneeded.
+
+Instructions for setup on Full Dev can be found at 
[Kerberos-setup.md](vagrant/Kerberos-setup.md).  These instructions can also be 
used for setting up KDC and testing.
+
+### Kerberos Without an MPack
+Using the MPack is preferred, but instructions for Kerberizing manually can be 
found at [Kerberos-manual-setup.md](Kerberos-manual-setup.md)
 
 ## TODO
 - Support Ubuntu deployments

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
----------------------------------------------------------------------
diff --git 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
index 25237f5..a4e82e2 100644
--- 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
+++ 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
@@ -177,9 +177,9 @@
 
         <component>
           <name>METRON_CLIENT</name>
-          <displayName>METRON Client</displayName>
+          <displayName>Metron Client</displayName>
           <category>CLIENT</category>
-          <cardinality>1+</cardinality>
+          <cardinality>0+</cardinality>
           <versionAdvertised>true</versionAdvertised>
           <commandScript>
             <script>scripts/metron_client.py</script>

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
----------------------------------------------------------------------
diff --git 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
index 4fda07b..136eade 100755
--- 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
+++ 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
@@ -21,6 +21,7 @@ from resource_management.core.logger import Logger
 from resource_management.core.resources.system import Execute, File
 
 import metron_service
+from metron_security import kinit
 
 # Wrap major operations and functionality in this class
 class EnrichmentCommands:
@@ -184,6 +185,11 @@ class EnrichmentCommands:
 
     def create_hbase_tables(self):
         Logger.info("Creating HBase Tables")
+        if self.__params.security_enabled:
+            kinit(self.__params.kinit_path_local,
+                  self.__params.hbase_keytab_path,
+                  self.__params.hbase_principal_name,
+                  execute_user=self.__params.hbase_user)
         cmd = "echo \"create '{0}','{1}'\" | hbase shell -n"
         add_enrichment_cmd = cmd.format(self.__params.enrichment_table, 
self.__params.enrichment_cf)
         Execute(add_enrichment_cmd,
@@ -208,6 +214,11 @@ class EnrichmentCommands:
 
     def set_hbase_acls(self):
         Logger.info("Setting HBase ACLs")
+        if self.__params.security_enabled:
+            kinit(self.__params.kinit_path_local,
+                  self.__params.hbase_keytab_path,
+                  self.__params.hbase_principal_name,
+                  execute_user=self.__params.hbase_user)
         cmd = "echo \"grant '{0}', 'RW', '{1}'\" | hbase shell -n"
         add_enrichment_acl_cmd = cmd.format(self.__params.metron_user, 
self.__params.enrichment_table)
         Execute(add_enrichment_acl_cmd,

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
----------------------------------------------------------------------
diff --git 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
index 65831c9..32fe0e9 100644
--- 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
+++ 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_client.py
@@ -46,4 +46,4 @@ class MetronClient(Script):
         raise ClientComponentHasNoStatus()
 
 if __name__ == "__main__":
-    MetronClient().execute()
\ No newline at end of file
+    MetronClient().execute()

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
----------------------------------------------------------------------
diff --git 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
index f2cb88d..be1654f 100644
--- 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
+++ 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
@@ -68,22 +68,6 @@ def get_running_topologies(params):
     Logger.info("Topologies: " + str(topologiesDict))
     return topologiesDict
 
-    try:
-        stormjson = json.loads(stdout)
-    except ValueError, e:
-        Logger.info('Stdout: ' + str(stdout))
-        Logger.info('Stderr: ' + str(stderr))
-        Logger.exception(str(e))
-        return {}
-
-    topologiesDict = {}
-
-    for topology in stormjson['topologies']:
-        topologiesDict[topology['name']] = topology['status']
-
-    Logger.info("Topologies: " + str(topologiesDict))
-    return topologiesDict
-
 
 def load_global_config(params):
     Logger.info('Create Metron Local Config Directory')

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
----------------------------------------------------------------------
diff --git 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
index 965502a..e4cae47 100755
--- 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
+++ 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py
@@ -113,11 +113,10 @@ class ParserCommands:
         if self.__params.security_enabled:
             # Append the extra configs needed for secured cluster.
             start_cmd_template = start_cmd_template + ' -e ~' + 
self.__params.metron_user + '/.storm/storm.config'
-            if self.__params.security_enabled:
-                metron_security.kinit(self.__params.kinit_path_local,
-                                      self.__params.metron_keytab_path,
-                                      self.__params.metron_principal_name,
-                                      execute_user=self.__params.metron_user)
+            metron_security.kinit(self.__params.kinit_path_local,
+                                  self.__params.metron_keytab_path,
+                                  self.__params.metron_principal_name,
+                                  execute_user=self.__params.metron_user)
         for parser in self.get_parser_list():
             Logger.info('Starting ' + parser)
             Execute(start_cmd_template.format(self.__params.metron_home,

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
----------------------------------------------------------------------
diff --git 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
index ff2498c..c0a047e 100644
--- 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
+++ 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/client_jaas.conf.j2
@@ -41,4 +41,4 @@ KafkaClient {
    useTicketCache=false
    serviceName="kafka"
    principal="{{metron_principal_name}}";
-};
\ No newline at end of file
+};

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
----------------------------------------------------------------------
diff --git 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
index caf1221..85fdf5c 100644
--- 
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
+++ 
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/storm.config.j2
@@ -19,4 +19,4 @@
 {
   "topology.worker.childopts" : 
"-Djava.security.auth.login.config={{metron_home}}/client_jaas.conf",
   "topology.auto-credentials" : {{topology_auto_credentials_double_quotes}}
-}
\ No newline at end of file
+}

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/ambari-storm-site-properties.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/ambari-storm-site-properties.png 
b/metron-deployment/readme-images/ambari-storm-site-properties.png
new file mode 100755
index 0000000..e0050a2
Binary files /dev/null and 
b/metron-deployment/readme-images/ambari-storm-site-properties.png differ

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/ambari-storm-site.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/ambari-storm-site.png 
b/metron-deployment/readme-images/ambari-storm-site.png
new file mode 100755
index 0000000..5ff2d24
Binary files /dev/null and 
b/metron-deployment/readme-images/ambari-storm-site.png differ

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/custom-storm-site-final.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/custom-storm-site-final.png 
b/metron-deployment/readme-images/custom-storm-site-final.png
new file mode 100755
index 0000000..9b383d5
Binary files /dev/null and 
b/metron-deployment/readme-images/custom-storm-site-final.png differ

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/enable-kerberos-configure-kerberos.png
----------------------------------------------------------------------
diff --git 
a/metron-deployment/readme-images/enable-kerberos-configure-kerberos.png 
b/metron-deployment/readme-images/enable-kerberos-configure-kerberos.png
new file mode 100755
index 0000000..212c64b
Binary files /dev/null and 
b/metron-deployment/readme-images/enable-kerberos-configure-kerberos.png differ

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/enable-kerberos-started.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/enable-kerberos-started.png 
b/metron-deployment/readme-images/enable-kerberos-started.png
new file mode 100755
index 0000000..96adb51
Binary files /dev/null and 
b/metron-deployment/readme-images/enable-kerberos-started.png differ

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/readme-images/enable-kerberos.png
----------------------------------------------------------------------
diff --git a/metron-deployment/readme-images/enable-kerberos.png 
b/metron-deployment/readme-images/enable-kerberos.png
new file mode 100755
index 0000000..bb46923
Binary files /dev/null and 
b/metron-deployment/readme-images/enable-kerberos.png differ

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/Kerberos-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/Kerberos-setup.md 
b/metron-deployment/vagrant/Kerberos-setup.md
index 27a56b0..f02cc5f 100644
--- a/metron-deployment/vagrant/Kerberos-setup.md
+++ b/metron-deployment/vagrant/Kerberos-setup.md
@@ -1,5 +1,8 @@
 # Setting Up Kerberos in Vagrant Full Dev
 **Note:** These are instructions for Kerberizing Metron Storm topologies from 
Kafka to Kafka. This does not cover the sensor connections or MAAS.
+General Kerberization notes can be found in the metron-deployment 
[README.md](../README.md)
+
+## Setup the KDC
 
 1. Build full dev and ssh into the machine
     ```
@@ -19,12 +22,7 @@
     export METRON_HOME="/usr/metron/${METRON_VERSION}"
     ```
 
-3. Stop all topologies - we will  restart them again once Kerberos has been 
enabled.
-    ```
-    for topology in bro snort enrichment indexing; do storm kill $topology; 
done
-    ```
-
-4. Setup Kerberos
+3. Setup Kerberos
     ```
     # Note: if you copy/paste this full set of commands, the kdb5_util command 
will not run as expected, so run the commands individually to ensure they all 
execute
     # set 'node1' to the correct host for your kdc
@@ -39,210 +37,46 @@
     chkconfig kadmin on
     ```
 
-5. Setup the admin and metron user principals. You'll kinit as the metron user 
when running topologies. Make sure to remember the passwords.
+4. Setup the admin user principal. You'll kinit as the metron user when 
running topologies. Make sure to remember the password.
     ```
     kadmin.local -q "addprinc admin/admin"
-    kadmin.local -q "addprinc metron"
-    ```
-
-6. Create the metron user HDFS home directory
-    ```
-    sudo -u hdfs hdfs dfs -mkdir /user/metron && \
-    sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
-    sudo -u hdfs hdfs dfs -chmod 770 /user/metron
-    ```
-
-7. In [Ambari](http://node1:8080), setup Storm to run with Kerberos and run 
worker jobs as the submitting user:
-
-    a. Add the following properties to custom storm-site:
     ```
-    
topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
-    
nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
-    supervisor.run.worker.as.user=true
-    ```
-
-    b. In the Storm config section in Ambari, choose “Add Property” under 
custom storm-site:
 
-    ![custom storm-site](readme-images/ambari-storm-site.png)
-
-    c. In the dialog window, choose the “bulk property add mode” toggle 
button and add the below values:
-
-    ![custom storm-site 
properties](readme-images/ambari-storm-site-properties.png)
-
-8. Kerberize the cluster via Ambari. More detailed documentation can be found 
[here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+## Ambari Setup
+1. Kerberize the cluster via Ambari. More detailed documentation can be found 
[here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
 
     a. For this exercise, choose existing MIT KDC (this is what we setup and 
installed in the previous steps.)
 
-    ![enable keberos](readme-images/enable-kerberos.png)
+    ![enable keberos](../readme-images/enable-kerberos.png)
 
-    ![enable keberos get started](readme-images/enable-kerberos-started.png)
+    ![enable keberos get started](../readme-images/enable-kerberos-started.png)
 
     b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal 
will end up as admin/ad...@example.com when testing the KDC. Use the password 
you entered during the step for adding the admin principal.
 
-    ![enable keberos 
configure](readme-images/enable-kerberos-configure-kerberos.png)
-
-    c. Click through to “Start and Test Services.” Let the cluster spin 
up, but don't worry about starting up Metron via Ambari - we're going to run 
the parsers manually against the rest of the Hadoop cluster Kerberized. The 
wizard will fail at starting Metron, but this is OK. Click “continue.” When 
you’re finished, the custom storm-site should look similar to the following:
+    ![enable keberos 
configure](../readme-images/enable-kerberos-configure-kerberos.png)
 
-    ![enable keberos configure](readme-images/custom-storm-site-final.png)
+    c. Click through to “Start and Test Services.” Let the cluster spin up.
 
-9. Setup Metron keytab
-    ```
-    kadmin.local -q "ktadd -k metron.headless.keytab met...@example.com" && \
-    cp metron.headless.keytab /etc/security/keytabs && \
-    chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
-    chmod 440 /etc/security/keytabs/metron.headless.keytab
-    ```
-
-10. Kinit with the metron user
+## Push Data
+1. Kinit with the metron user
     ```
     kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
     ```
 
-11. First create any additional Kafka topics you will need. We need to create 
the topics before adding the required ACLs. The current full dev installation 
will deploy bro, snort, enrichments, and indexing only. e.g.
-    ```
-    ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 
--create --topic yaf --partitions 1 --replication-factor 1
-    ```
-
-12. Setup Kafka ACLs for the topics
-    ```
-    export KERB_USER=metron
-    for topic in bro enrichments indexing snort; do
-        ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--topic ${topic}
-    done
-    ```
-
-13. Setup Kafka ACLs for the consumer groups
-    ```
-    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--group bro_parser
-    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--group snort_parser
-    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--group yaf_parser
-    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--group enrichments
-    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--group indexing
-    ```
-
-14. Add metron user to the Kafka cluster ACL
-    ```
-    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} 
--cluster kafka-cluster
-    ```
-
-15. We also need to grant permissions to the HBase tables. Kinit as the hbase 
user and add ACLs for metron.
-    ```
-    kinit -kt /etc/security/keytabs/hbase.headless.keytab 
hbase-metron_clus...@example.com
-    echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
-    echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
-    ```
-
-16. Create a “.storm” directory in the metron user’s home directory and 
switch to that directory.
-    ```
-    su metron
-    mkdir ~/.storm
-    cd ~/.storm
-    ```
-
-17. Create a custom client jaas file. This should look identical to the Storm 
client jaas file located in /etc/storm/conf/client_jaas.conf except for the 
addition of a Client stanza. The Client stanza is used for Zookeeper. All 
quotes and semicolons are necessary.
-    ```
-    cat << EOF > client_jaas.conf
-    StormClient {
-        com.sun.security.auth.module.Krb5LoginModule required
-        useTicketCache=true
-        renewTicket=true
-        serviceName="nimbus";
-    };
-    Client {
-        com.sun.security.auth.module.Krb5LoginModule required
-        useKeyTab=true
-        keyTab="/etc/security/keytabs/metron.headless.keytab"
-        storeKey=true
-        useTicketCache=false
-        serviceName="zookeeper"
-        principal="met...@example.com";
-    };
-    KafkaClient {
-        com.sun.security.auth.module.Krb5LoginModule required
-        useKeyTab=true
-        keyTab="/etc/security/keytabs/metron.headless.keytab"
-        storeKey=true
-        useTicketCache=false
-        serviceName="kafka"
-        principal="met...@example.com";
-    };
-    EOF
-    ```
-
-18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts 
accordingly.
-    ```
-    cat << EOF > storm.yaml
-    nimbus.seeds : ['node1']
-    java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
-    storm.thrift.transport : 
'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
-    EOF
-    ```
-
-19. Create an auxiliary storm configuration json file in the metron user’s 
home directory. Note the login config option in the file points to our custom 
client_jaas.conf.
-    ```
-    cat << EOF > ~/storm-config.json
-    {
-        "topology.worker.childopts" : 
"-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
-    }
-    EOF
-    ```
-
-20. Setup enrichment and indexing.
-
-    a. Modify enrichment.properties as root located at 
`${METRON_HOME}/config/enrichment.properties`
-    ```
-    if [[ $EUID -ne 0 ]]; then
-        echo -e "\nERROR:\tYou must be root to run these commands.  You may 
need to type exit."
-    else
-        sed -i 
's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' 
${METRON_HOME}/config/enrichment.properties
-        sed -i 
's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/'
 ${METRON_HOME}/config/enrichment.properties
-    fi
-    ```
-
-    b. Modify elasticsearch.properties as root located at 
`${METRON_HOME}/config/elasticsearch.properties`
-    ```
-    if [[ $EUID -ne 0 ]]; then
-        echo -e "\nERROR:\tYou must be root to run these commands.  You may 
need to type exit."
-    else
-        sed -i 
's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' 
${METRON_HOME}/config/elasticsearch.properties
-        sed -i 
's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/'
 ${METRON_HOME}/config/elasticsearch.properties
-    fi
-    ```
-
-21. Kinit with the metron user again
-    ```
-    su metron
-    cd
-    kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
-    ```
-
-22. Restart the parser topologies. Be sure to pass in the new parameter, 
“-ksp” or “--kafka_security_protocol.” Run this from the metron home 
directory.
-    ```
-    for parser in bro snort; do
-        ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s 
${parser} -ksp SASL_PLAINTEXT -e storm-config.json
-    done
-    ```
-
-23. Now restart the enrichment and indexing topologies.
-    ```
-    ${METRON_HOME}/bin/start_enrichment_topology.sh
-    ${METRON_HOME}/bin/start_elasticsearch_topology.sh
-    ```
-
-24. Push some sample data to one of the parser topics. E.g for bro we took raw 
data from 
[incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
+2. Push some sample data to one of the parser topics. E.g for bro we took raw 
data from 
[incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
     ```
     cat sample-bro.txt | 
${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list 
${BROKERLIST}:6667 --security-protocol SASL_PLAINTEXT --topic bro
     ```
 
-25. Wait a few moments for data to flow through the system and then check for 
data in the Elasticsearch indexes. Replace bro with whichever parser type 
you’ve chosen.
+3. Wait a few moments for data to flow through the system and then check for 
data in the Elasticsearch indexes. Replace bro with whichever parser type 
you’ve chosen.
     ```
     curl -XGET "${ZOOKEEPER}:9200/bro*/_search"
     curl -XGET "${ZOOKEEPER}:9200/bro*/_count"
     ```
 
-26. You should have data flowing from the parsers all the way through to the 
indexes. This completes the Kerberization instructions
+4. You should have data flowing from the parsers all the way through to the 
indexes. This completes the Kerberization instructions
 
-### Other useful commands:
+### Other useful commands
 #### Kerberos
 Unsure of your Kerberos principal associated with a keytab? There are a couple 
ways to get this. One is via the list of principals that Ambari provides via 
downloadable csv. If you didn’t download this list, you can also check the 
principal manually by running the following against the keytab.
 ```

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png
----------------------------------------------------------------------
diff --git 
a/metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png 
b/metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png
deleted file mode 100755
index e0050a2..0000000
Binary files 
a/metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png and 
/dev/null differ

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/ambari-storm-site.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/ambari-storm-site.png 
b/metron-deployment/vagrant/readme-images/ambari-storm-site.png
deleted file mode 100755
index 5ff2d24..0000000
Binary files a/metron-deployment/vagrant/readme-images/ambari-storm-site.png 
and /dev/null differ

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/custom-storm-site-final.png
----------------------------------------------------------------------
diff --git 
a/metron-deployment/vagrant/readme-images/custom-storm-site-final.png 
b/metron-deployment/vagrant/readme-images/custom-storm-site-final.png
deleted file mode 100755
index 9b383d5..0000000
Binary files 
a/metron-deployment/vagrant/readme-images/custom-storm-site-final.png and 
/dev/null differ

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png
----------------------------------------------------------------------
diff --git 
a/metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png
 
b/metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png
deleted file mode 100755
index 212c64b..0000000
Binary files 
a/metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png
 and /dev/null differ

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/enable-kerberos-started.png
----------------------------------------------------------------------
diff --git 
a/metron-deployment/vagrant/readme-images/enable-kerberos-started.png 
b/metron-deployment/vagrant/readme-images/enable-kerberos-started.png
deleted file mode 100755
index 96adb51..0000000
Binary files 
a/metron-deployment/vagrant/readme-images/enable-kerberos-started.png and 
/dev/null differ

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/41b5b105/metron-deployment/vagrant/readme-images/enable-kerberos.png
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/readme-images/enable-kerberos.png 
b/metron-deployment/vagrant/readme-images/enable-kerberos.png
deleted file mode 100755
index bb46923..0000000
Binary files a/metron-deployment/vagrant/readme-images/enable-kerberos.png and 
/dev/null differ


Reply via email to