This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 345f991 METRON-2233 Deprecate CentOS 6 Development Environment
(nickwallen) closes apache/metron#1542
345f991 is described below
commit 345f99157e4ed3fa3730909c53eea8f09a3167cf
Author: nickwallen <nickal...@apache.org>
AuthorDate: Wed Oct 30 12:35:00 2019 -0400
METRON-2233 Deprecate CentOS 6 Development Environment (nickwallen) closes
apache/metron#1542
---
README.md | 2 +-
dev-utilities/release-utils/metron-rc-check | 2 +-
metron-contrib/metron-docker/README.md | 2 +-
metron-deployment/Kerberos-manual-setup.md | 12 +--
metron-deployment/README.md | 2 +-
.../ansible/playbooks/ambari_install.yml | 2 +-
.../ansible/playbooks/metron_install.yml | 2 +-
.../ansible/playbooks/sensor_install.yml | 2 +-
.../ansible/playbooks/solr_install.yml | 2 +-
.../ansible/roles/ambari_common/defaults/main.yml | 1 -
.../ambari_common/tasks/ambari-repo-CentOS-6.yml | 22 -----
.../ambari_common/tasks/firewall-CentOS-6.yml | 20 ----
.../roles/ambari_common/tasks/nodejs-CentOS-6.yml | 22 -----
.../ansible/roles/sensor-stubs/README.md | 2 +-
metron-deployment/development/README.md | 4 +-
metron-deployment/development/centos6/.gitignore | 1 -
metron-deployment/development/centos6/Vagrantfile | 107 ---------------------
metron-deployment/development/centos6/ansible.cfg | 28 ------
.../centos6/ansible/inventory/group_vars/all | 86 -----------------
.../development/centos6/ansible/inventory/hosts | 47 ---------
.../development/centos6/ansible/playbook.yml | 24 -----
.../development/{centos6 => centos7}/README.md | 30 ++++--
.../packaging/docker/rpm-docker/Dockerfile | 2 +-
metron-interface/metron-rest/README.md | 4 +-
.../metron-solr/metron-solr-common/README.md | 10 +-
25 files changed, 48 insertions(+), 390 deletions(-)
diff --git a/README.md b/README.md
index 1f4e030..a3bc14e 100644
--- a/README.md
+++ b/README.md
@@ -83,7 +83,7 @@ Option 3 is more likely to have the latest code.
# Getting Started
-To start exploring the capabilities of Apache Metron [follow these
instructions to launch Metron in a single-node
VM](metron-deployment/development/centos6) on your own hardware.
+To start exploring the capabilities of Apache Metron [follow these
instructions to launch Metron in a single-node
VM](metron-deployment/development/centos7) on your own hardware.
# Building Metron
diff --git a/dev-utilities/release-utils/metron-rc-check
b/dev-utilities/release-utils/metron-rc-check
index 5b3a5d9..66752e1 100755
--- a/dev-utilities/release-utils/metron-rc-check
+++ b/dev-utilities/release-utils/metron-rc-check
@@ -274,7 +274,7 @@ echo ""
read -p " run vagrant full_dev? [yN] " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
- cd
"$WORK/apache-metron_${METRON_VERSION}-$RC/metron-deployment/development/centos6"
|| exit 1
+ cd
"$WORK/apache-metron_${METRON_VERSION}-$RC/metron-deployment/development/centos7"
|| exit 1
if [[ ${DID_BUILD} -ne 1 ]]; then
vagrant up
else
diff --git a/metron-contrib/metron-docker/README.md
b/metron-contrib/metron-docker/README.md
index 761cc6e..0dcf739 100644
--- a/metron-contrib/metron-docker/README.md
+++ b/metron-contrib/metron-docker/README.md
@@ -19,7 +19,7 @@ limitations under the License.
Metron Docker is a [Docker Compose](https://docs.docker.com/compose/overview/)
application that is intended only for development and integration testing of
Metron. These images can quickly spin-up the underlying components on which
Apache Metron runs.
-None of the core Metron components are setup or launched automatically with
these Docker images. You will need to manually setup and start the Metron
components that you require. You should not expect to see telemetry being
parsed, enriched, or indexed. If you are looking to try-out, experiment or
demo Metron capabilities on a single node, then the [Vagrant-driven
VM](../../metron-deployment/development/centos6) is what you need. Use this
instead of Vagrant when:
+None of the core Metron components are setup or launched automatically with
these Docker images. You will need to manually setup and start the Metron
components that you require. You should not expect to see telemetry being
parsed, enriched, or indexed. If you are looking to try-out, experiment or
demo Metron capabilities on a single node, then the [Vagrant-driven
VM](../../metron-deployment/development/centos7) is what you need. Use this
instead of Vagrant when:
- You want an environment that can be built and spun up quickly
- You need to frequently rebuild and restart services
diff --git a/metron-deployment/Kerberos-manual-setup.md
b/metron-deployment/Kerberos-manual-setup.md
index aeedd95..2340eca 100644
--- a/metron-deployment/Kerberos-manual-setup.md
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -36,7 +36,7 @@ This document provides instructions for kerberizing Metron's
Vagrant-based devel
Setup
-----
-1. Deploy the [development environment.](development/centos6/README.md).
+1. Deploy the [development environment.](development/centos7/README.md).
1. Export the following environment variables. These need to be set for the
remainder of the instructions. Replace `node1` with the appropriate hosts, if
you are running Metron anywhere other than Vagrant.
@@ -88,12 +88,12 @@ Setup a KDC
```
# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/
-
+
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
-
+
[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 24h
@@ -103,13 +103,13 @@ Setup a KDC
pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
default_realm = EXAMPLE.COM
default_ccache_name = KEYRING:persistent:%{uid}
-
+
[realms]
EXAMPLE.COM = {
kdc = node1
admin_server = node1
}
-
+
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM
@@ -706,5 +706,3 @@ Execute it like the following example:
su - metron
python $METRON_HOME/bin/tgt_renew.py node1:8744 metron
```
-
-
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index 73cf602..7613fa7 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -67,7 +67,7 @@ Running Metron within the resource constraints of a single VM
is incredibly chal
#### How?
-To deploy Metron in a VM running on your computer, follow the instructions at
[development/centos6](development/centos6).
+To deploy Metron in a VM running on your computer, follow the instructions at
[development/centos7](development/centos7).
#### How do I address services crashing when running Metron on a single VM?
diff --git a/metron-deployment/ansible/playbooks/ambari_install.yml
b/metron-deployment/ansible/playbooks/ambari_install.yml
index 5b802b3..ca11cee 100644
--- a/metron-deployment/ansible/playbooks/ambari_install.yml
+++ b/metron-deployment/ansible/playbooks/ambari_install.yml
@@ -25,7 +25,7 @@
- hosts: packer
become: true
tasks:
- - include_vars: ../../development/centos6/ansible/inventory/group_vars/all
+ - include_vars: ../../development/centos7/ansible/inventory/group_vars/all
tags:
- packer
diff --git a/metron-deployment/ansible/playbooks/metron_install.yml
b/metron-deployment/ansible/playbooks/metron_install.yml
index 56f7a7a..30c0692 100644
--- a/metron-deployment/ansible/playbooks/metron_install.yml
+++ b/metron-deployment/ansible/playbooks/metron_install.yml
@@ -25,7 +25,7 @@
- hosts: packer
become: true
tasks:
- - include_vars: ../../development/centos6/ansible/inventory/group_vars/all
+ - include_vars: ../../development/centos7/ansible/inventory/group_vars/all
tags:
- packer
diff --git a/metron-deployment/ansible/playbooks/sensor_install.yml
b/metron-deployment/ansible/playbooks/sensor_install.yml
index a683ccb..c425539 100644
--- a/metron-deployment/ansible/playbooks/sensor_install.yml
+++ b/metron-deployment/ansible/playbooks/sensor_install.yml
@@ -25,7 +25,7 @@
- hosts: packer
become: true
tasks:
- - include_vars: ../../development/centos6/ansible/inventory/group_vars/all
+ - include_vars: ../../development/centos7/ansible/inventory/group_vars/all
tags:
- packer
diff --git a/metron-deployment/ansible/playbooks/solr_install.yml
b/metron-deployment/ansible/playbooks/solr_install.yml
index 464ec80..80fdf1f 100644
--- a/metron-deployment/ansible/playbooks/solr_install.yml
+++ b/metron-deployment/ansible/playbooks/solr_install.yml
@@ -25,7 +25,7 @@
- hosts: packer
become: true
tasks:
- - include_vars: ../../development/centos6/ansible/inventory/group_vars/all
+ - include_vars: ../../development/centos7/ansible/inventory/group_vars/all
tags:
- packer
diff --git a/metron-deployment/ansible/roles/ambari_common/defaults/main.yml
b/metron-deployment/ansible/roles/ambari_common/defaults/main.yml
index 344638e..8c950da 100644
--- a/metron-deployment/ansible/roles/ambari_common/defaults/main.yml
+++ b/metron-deployment/ansible/roles/ambari_common/defaults/main.yml
@@ -17,6 +17,5 @@
---
hadoop_logrotate_frequency: daily
hadoop_logrotate_retention: 30
-centos6_ambari_install_url:
http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.6.1.0/ambari.repo
centos7_ambari_install_url:
http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.0/ambari.repo
ubuntu_ambari_repo:
http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.6.1.0
diff --git
a/metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-CentOS-6.yml
b/metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-CentOS-6.yml
deleted file mode 100644
index 7e6d9d2..0000000
---
a/metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-CentOS-6.yml
+++ /dev/null
@@ -1,22 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: Setup Ambari repo on CentOS
- get_url: url="{{ centos6_ambari_install_url }}"
dest=/etc/yum.repos.d/ambari.repo
-
-- name: Update package cache on CentOS
- yum: name=* update_cache=yes
diff --git
a/metron-deployment/ansible/roles/ambari_common/tasks/firewall-CentOS-6.yml
b/metron-deployment/ansible/roles/ambari_common/tasks/firewall-CentOS-6.yml
deleted file mode 100644
index 8f01e41..0000000
--- a/metron-deployment/ansible/roles/ambari_common/tasks/firewall-CentOS-6.yml
+++ /dev/null
@@ -1,20 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: Stop iptables on CentOS 6
- ignore_errors: yes
- service: name=iptables state=stopped enabled=no
diff --git
a/metron-deployment/ansible/roles/ambari_common/tasks/nodejs-CentOS-6.yml
b/metron-deployment/ansible/roles/ambari_common/tasks/nodejs-CentOS-6.yml
deleted file mode 100644
index 8ea7363..0000000
--- a/metron-deployment/ansible/roles/ambari_common/tasks/nodejs-CentOS-6.yml
+++ /dev/null
@@ -1,22 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: Install Node.js repo on CentOS 6
- shell: curl -sL https://rpm.nodesource.com/setup_6.x | bash -
- args:
- warn: false
- become: true
diff --git a/metron-deployment/ansible/roles/sensor-stubs/README.md
b/metron-deployment/ansible/roles/sensor-stubs/README.md
index 015f1d1..1b8c677 100644
--- a/metron-deployment/ansible/roles/sensor-stubs/README.md
+++ b/metron-deployment/ansible/roles/sensor-stubs/README.md
@@ -90,7 +90,7 @@ Stopping sensor-stubs...
The default behavior can be changed by skipping the `sensor-stubs` flag and
including the `sensors` flag. For example, to deploy "Full Dev" with the
original sensors run the following command.
```
-cd metron-deployment/development/centos6
+cd metron-deployment/development/centos7
vagrant --ansible-skip-tags="sensor-stubs,solr" up
```
diff --git a/metron-deployment/development/README.md
b/metron-deployment/development/README.md
index 448b53f..2aee8bf 100644
--- a/metron-deployment/development/README.md
+++ b/metron-deployment/development/README.md
@@ -19,7 +19,7 @@ limitations under the License.
This directory contains environments useful for Metron developers. These
environments are not intended for proof-of-concept, testing, or production use.
These are extremely resource constrained and cannot support anything beyond
the most basic work loads.
-* Metron running on CentOS 6
+* Metron running on CentOS 7
* Metron running on Ubuntu 14
* Fastcapa
@@ -51,7 +51,7 @@ To setup this up, start full dev.
* Bind User = uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
* LDAP Truststore is empty
* LDAP Truststore Password is empty
-
+
* In the Knox configuration, go to "Advanced users-ldif". We have a custom
ldif file "knox-demo-ldap.ldif" in "metron-deployment/development" that
contains a customized variant of the users and groups defined here. Replace the
default ldif configuration with the contents of "knox-demo-ldap.ldif"
* Start the Demo LDAP (In Knox, "Service Actions -> Start Demo LDAP)
* In Metron's configs, we're going to make two changes
diff --git a/metron-deployment/development/centos6/.gitignore
b/metron-deployment/development/centos6/.gitignore
deleted file mode 100644
index 8000dd9..0000000
--- a/metron-deployment/development/centos6/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-.vagrant
diff --git a/metron-deployment/development/centos6/Vagrantfile
b/metron-deployment/development/centos6/Vagrantfile
deleted file mode 100644
index 21ca2bd..0000000
--- a/metron-deployment/development/centos6/Vagrantfile
+++ /dev/null
@@ -1,107 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-require 'getoptlong'
-
-ansibleTags=''
-ansibleSkipTags='sensors'
-
-begin
- opts = GetoptLong.new(
- [ '--ansible-tags', GetoptLong::OPTIONAL_ARGUMENT ],
- [ '--ansible-skip-tags', GetoptLong::OPTIONAL_ARGUMENT ]
- )
-
- opts.quiet = true
-
- opts.each do |opt, arg|
- case opt
- when '--ansible-tags'
- ansibleTags=arg
- when '--ansible-skip-tags'
- ansibleSkipTags=arg
- end
- end
-rescue Exception => ignored
-#Ignore to allow other opts to be passed to Vagrant
-end
-
-puts " Running with ansible-tags: " + ansibleTags.split(",").to_s if
ansibleTags != ''
-puts " Running with ansible-skip-tags: " + ansibleSkipTags.split(",").to_s if
ansibleSkipTags != ''
-
-hosts = [{
- hostname: "node1",
- ip: "192.168.66.121",
- memory: "8192",
- cpus: 4,
- promisc: 2 # enables promisc on the 'Nth' network interface
-}]
-
-Vagrant.configure(2) do |config|
-
- # host built on centos 6
- config.vm.box = "centos/6"
- config.ssh.insert_key = true
-
- # enable the hostmanager plugin
- config.hostmanager.enabled = true
- config.hostmanager.manage_host = true
-
- # enable vagrant cachier if present
- if Vagrant.has_plugin?("vagrant-cachier")
- config.cache.enable :yum
- config.cache.scope = :box
-
- config.cache.synced_folder_opts = {
- type: :nfs,
- mount_options: ['rw', 'vers=3', 'tcp', 'nolock']
- }
- end
-
- # host definition
- hosts.each_with_index do |host, index|
- config.vm.define host[:hostname] do |node|
-
- # host settings
- node.vm.hostname = host[:hostname]
- node.vm.network "private_network", ip: host[:ip]
-
- # vm settings
- node.vm.provider "virtualbox" do |vb|
- vb.memory = host[:memory]
- vb.cpus = host[:cpus]
-
- # enable promisc mode on the network interface
- if host.has_key?(:promisc)
- vb.customize ["modifyvm", :id, "--nicpromisc#{host[:promisc]}",
"allow-all"]
- end
- # disable audio, so that the vm doesn't capture the sound / mic
- vb.customize ["modifyvm", :id, "--audio", "none"]
- end
- end
- end
-
- # provision the host with ansible
- config.vm.provision :ansible do |ansible|
- ansible.playbook = "ansible/playbook.yml"
- ansible.become = true
- ansible.tags = ansibleTags.split(",") if ansibleTags != ''
- ansible.skip_tags = ansibleSkipTags.split(",") if ansibleSkipTags != ''
- ansible.inventory_path = "ansible/inventory"
- ansible.compatibility_mode = "auto"
- ansible.raw_arguments = Shellwords.shellsplit(ENV['ANSIBLE_ARGS']) if
ENV['ANSIBLE_ARGS']
- end
-end
diff --git a/metron-deployment/development/centos6/ansible.cfg
b/metron-deployment/development/centos6/ansible.cfg
deleted file mode 100644
index 70ac5de..0000000
--- a/metron-deployment/development/centos6/ansible.cfg
+++ /dev/null
@@ -1,28 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-[defaults]
-host_key_checking = false
-library = ../../ansible/extra_modules
-roles_path = ../../ansible/roles
-pipelining = True
-log_path = ./ansible.log
-callback_plugins = ../../ansible/callback_plugins
-callback_whitelist = profile_tasks
-
-# fix for "ssh throws 'unix domain socket too long' " problem
-[ssh_connection]
-control_path = %(directory)s/%%h-%%p-%%r
diff --git
a/metron-deployment/development/centos6/ansible/inventory/group_vars/all
b/metron-deployment/development/centos6/ansible/inventory/group_vars/all
deleted file mode 100644
index dde70e0..0000000
--- a/metron-deployment/development/centos6/ansible/inventory/group_vars/all
+++ /dev/null
@@ -1,86 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# only need to build the RPMs for CentOS
-metron_build_packages_cmd: "shell cd {{ metron_build_dir }}/metron-deployment
&& mvn clean package -DskipTests -Pbuild-rpms"
-
-# ambari
-ambari_host: "{{ groups.ambari_master[0] }}"
-hdp_host_group: "{{ groups.ambari_slave }}"
-ambari_port: 8080
-ambari_user: admin
-ambari_password: admin
-cluster_type: single_node_vm
-ambari_server_mem: 512
-java_home: /usr/jdk64/jdk1.8.0_77
-
-# hbase
-pcap_hbase_table: pcap
-tracker_hbase_table: access_tracker
-threatintel_hbase_table: threatintel
-enrichment_hbase_table: enrichment
-
-# metron
-metron_version: 0.7.2
-metron_directory: /usr/metron/{{ metron_version }}
-bro_version: "2.5.5"
-fixbuf_version: "1.7.1"
-yaf_version: "2.8.0"
-daq_version: "2.0.6-1"
-pycapa_repo: "https://github.com/OpenSOC/pycapa.git";
-pycapa_home: "/opt/pycapa"
-snort_version: "2.9.8.0-1"
-snort_alert_csv_path: "/var/log/snort/alert.csv"
-threat_intel_bulk_load: False
-elasticsearch_hosts: "{{ groups.search | join(',') }}"
-
-# data directories - only required to override defaults
-zookeeper_data_dir: "/data1/hadoop/zookeeper"
-namenode_checkpoint_dir: "/data1/hadoop/hdfs/namesecondary"
-namenode_name_dir: "/data1/hadoop/hdfs/namenode"
-datanode_data_dir: "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data"
-journalnode_edits_dir: "/data1/hadoop/hdfs/journalnode"
-nodemanager_local_dirs: "/data1/hadoop/yarn/local"
-timeline_ldb_store_path: "/data1/hadoop/yarn/timeline"
-timeline_ldb_state_path: "/data1/hadoop/yarn/timeline"
-nodemanager_log_dirs: "/data1/hadoop/yarn/log"
-jhs_recovery_store_ldb_path: "/data1/hadoop/mapreduce/jhs"
-storm_local_dir: "/data1/hadoop/storm"
-kafka_log_dirs: "/data1/kafka-log"
-elasticsearch_data_dir: "/data1/elasticsearch,/data2/elasticsearch"
-
-# sensors
-sensor_test_mode: True
-install_pycapa: False
-install_bro: True
-install_snort: True
-install_yaf: False
-install_pcap_replay: True
-sniff_interface: eth1
-pcap_replay_interface: "{{ sniff_interface }}"
-pcapservice_port: 8081
-
-# search
-install_elasticsearch: True
-install_solr: False
-solr_collection_name: Metron
-solr_number_shards: 1
-solr_replication_factor: 1
-elasticsearch_transport_port: 9300
-## The elasticsearch_network_interface must be in the form specified for
Elasticsearch, with leading and trailing underscores.
-elasticsearch_network_interface: _eth1_
-elasticsearch_web_port: 9200
diff --git a/metron-deployment/development/centos6/ansible/inventory/hosts
b/metron-deployment/development/centos6/ansible/inventory/hosts
deleted file mode 100644
index 9bd9ea1..0000000
--- a/metron-deployment/development/centos6/ansible/inventory/hosts
+++ /dev/null
@@ -1,47 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-[ambari_master]
-node1
-
-[ambari_slave]
-node1
-
-[metron]
-node1
-
-[search]
-node1
-
-[sensors]
-node1
-
-[pcap_server]
-node1
-
-[web]
-node1
-
-[zeppelin]
-node1
-
-[monit:children]
-sensors
-pcap_server
-
-[local]
-127.0.0.1
diff --git a/metron-deployment/development/centos6/ansible/playbook.yml
b/metron-deployment/development/centos6/ansible/playbook.yml
deleted file mode 100644
index 433d3b5..0000000
--- a/metron-deployment/development/centos6/ansible/playbook.yml
+++ /dev/null
@@ -1,24 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- hosts: all
- roles:
- - { role: libselinux-python, tags: libselinux-python }
- - { role: enable-swap, tags: enable-swap }
- - { role: enable-remote-ssh, tags: enable-remote-ssh }
-
-- import_playbook: ../../../ansible/playbooks/metron_full_install.yml
diff --git a/metron-deployment/development/centos6/README.md
b/metron-deployment/development/centos7/README.md
similarity index 81%
rename from metron-deployment/development/centos6/README.md
rename to metron-deployment/development/centos7/README.md
index 62b4aa4..4bcd019 100644
--- a/metron-deployment/development/centos6/README.md
+++ b/metron-deployment/development/centos7/README.md
@@ -15,10 +15,10 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-Metron on CentOS 6
+Metron on CentOS 7
==================
-This project fully automates the provisioning and deployment of Apache Metron
and all necessary prerequisites on a single, virtualized host running CentOS 6.
+This project fully automates the provisioning and deployment of Apache Metron
and all necessary prerequisites on a single, virtualized host running CentOS 7.
Metron is composed of many components and installing all of these on a single
host, especially a virtualized one, will greatly stress the resources of the
host. The host will require at least 8 GB of RAM and a fair amount of
patience. It is highly recommended that you shut down all unnecessary services.
@@ -68,7 +68,7 @@ Any platform that supports these tools is suitable, but the
following instructio
1. Deploy Metron
```
- cd metron-deployment/development/centos6
+ cd metron-deployment/development/centos7
vagrant up
```
@@ -128,7 +128,24 @@ Tags are listed in the playbooks. Here are some
frequently used tags:
#### Sensors
-By default, the Metron development environment uses sensor stubs to mimic the
behavior of the full sensors. This is done because the full sensors take a
significant amount of time and CPU to build, install, and run.
+By default, the Metron development environment uses stubs to mimic the
behavior of running the full, demo sensor suit. For example, rather than
running an instance of Zeek and feeding it raw pcap, we simply publish a fixed
set of Zeek's output. This is done because the full, demo sensors take a
significant amount of time and CPU to build, install, and run.
+
+A separate Systemd service is installed for each demo sensor. This allows
each sensor to be started or stopped independently. Start the demo sensor stubs
by running the following commands.
+```
+systemctl start sensor-stubs-bro
+systemctl start sensor-stubs-snort
+systemctl start sensor-stubs-yaf
+```
+
+Stop the demo sensor stubs by running the following commands.
+```
+systemctl stop sensor-stubs-bro
+systemctl stop sensor-stubs-snort
+systemctl stop sensor-stubs-yaf
+```
+
+
+##### Full Sensors
From time to time you may want to install the full sensors for testing (see
the specifics of what that means
[here](../../ansible/playbooks/sensor_install.yml)). This can be done by
running the following command:
@@ -136,7 +153,9 @@ From time to time you may want to install the full sensors
for testing (see the
vagrant --ansible-skip-tags="sensor-stubs" up
```
-This will skip only the `sensor-stubs` tag, allowing the ansible roles with
the `sensors` tag to be run. This provisions the full sensors in a 'testing
mode' so that they are more active, and thus more useful for testing (more
details on that [here](../../ansible/roles/sensor-test-mode/)). **However**,
when vagrant completes the sensors will NOT be running. In order to start the
sensors and simulate traffic through them (which will create a fair amount of
load on your test system), co [...]
+This will skip only the `sensor-stubs` tag, allowing the ansible roles with
the `sensors` tag to be run. This provisions the full sensors in a 'testing
mode' so that they are more active, and thus more useful for testing (more
details on that [here](../../ansible/roles/sensor-test-mode/)).
+
+**However**, when vagrant completes the sensors will NOT be running. In order
to start the sensors and simulate traffic through them (which will create a
fair amount of load on your test system), complete the below steps:
```
vagrant ssh
@@ -146,4 +165,3 @@ service yaf restart
service snortd restart
service snort-producer restart
```
-
diff --git a/metron-deployment/packaging/docker/rpm-docker/Dockerfile
b/metron-deployment/packaging/docker/rpm-docker/Dockerfile
index f151678..e0c7b5f 100644
--- a/metron-deployment/packaging/docker/rpm-docker/Dockerfile
+++ b/metron-deployment/packaging/docker/rpm-docker/Dockerfile
@@ -15,7 +15,7 @@
# limitations under the License.
#
-FROM centos:centos6
+FROM centos:centos7
RUN yum install -y \
tar \
diff --git a/metron-interface/metron-rest/README.md
b/metron-interface/metron-rest/README.md
index a5b0e2e..c4dd028 100644
--- a/metron-interface/metron-rest/README.md
+++ b/metron-interface/metron-rest/README.md
@@ -1099,7 +1099,7 @@ The metron-rest application will be available at
http://localhost:8080/swagger-u
### Full Dev
-Start the [development
environment](../../metron-deployment/development/centos6). Build the
metron-rest module and start it with the Spring Boot Maven plugin:
+Start the [development
environment](../../metron-deployment/development/centos7). Build the
metron-rest module and start it with the Spring Boot Maven plugin:
```
mvn clean package
mvn spring-boot:run -Drun.profiles=vagrant,dev
@@ -1138,7 +1138,7 @@ The following instructions are for Intellij.
1. Using the Maven Projects tab, set check the `local-dev` profile in the
`Profiles` section:
-1. Start the `REST` Configuration in Debug mode using the Debug button in the
top right or the `Run > Debug 'REST'` menu item
+1. Start the `REST` Configuration in Debug mode using the Debug button in the
top right or the `Run > Debug 'REST'` menu item
The REST application should now available at
`http://localhost:8080/swagger-ui.html`.
diff --git a/metron-platform/metron-solr/metron-solr-common/README.md
b/metron-platform/metron-solr/metron-solr-common/README.md
index cf37589..88f7437 100644
--- a/metron-platform/metron-solr/metron-solr-common/README.md
+++ b/metron-platform/metron-solr/metron-solr-common/README.md
@@ -75,7 +75,7 @@ via the global config. The following settings are possible
as part of the globa
## Installing
-Solr is installed in the [full dev environment for
CentOS](../../metron-deployment/development/centos6) by default but is not
started initially. Navigate to `$METRON_HOME/bin`
+Solr is installed in the [full dev environment for
CentOS](../../metron-deployment/development/centos7) by default but is not
started initially. Navigate to `$METRON_HOME/bin`
and start Solr Cloud by running `start_solr.sh`.
Metron's Ambari MPack installs several scripts in `$METRON_HOME/bin` that can
be used to manage Solr. A script is also provided for installing Solr Cloud
outside of full dev.
@@ -88,7 +88,7 @@ The script performs the following tasks
Note: for details on setting up Solr Cloud in production mode, see
https://lucene.apache.org/solr/guide/7_4/taking-solr-to-production.html
-Navigate to `$METRON_HOME/bin` and spin up Solr Cloud by running
`install_solr.sh`. After running this script,
+Navigate to `$METRON_HOME/bin` and spin up Solr Cloud by running
`install_solr.sh`. After running this script,
Elasticsearch and Kibana will have been stopped and you should now have an
instance of Solr Cloud up and running at http://localhost:8983/solr/#/~cloud.
This manner of starting Solr
will also spin up an embedded Zookeeper instance at port 9983. More
information can be found
[here](https://lucene.apache.org/solr/guide/7_4/getting-started-with-solrcloud.html)
@@ -106,7 +106,7 @@ Elasticsearch is the real-time store used by default in
Metron. Solr can be ena
This will automatically create collections for the schemas shipped with Metron:
-* bro
+* bro
* snort
* yaf
* error (used internally by Metron)
@@ -144,7 +144,7 @@ Convenience scripts are provided with Metron to create and
delete collections.
# Path to the zookeeper node used by Solr
export ZOOKEEPER=node1:9983
# Set to true if Kerberos is enabled
-export SECURITY_ENABLED=true
+export SECURITY_ENABLED=true
```
The scripts can then be called directly with the collection name as the first
argument . For example, to create the bro collection:
```
@@ -163,4 +163,4 @@ The `create_collection.sh` script depends on schemas
installed in `$METRON_HOME/
* error
Additional schemas should be installed in that location if using the
`create_collection.sh` script. Any collection can be deleted with the
`delete_collection.sh` script.
-These scripts use the [Solr Collection
API](http://lucene.apache.org/solr/guide/7_4/collections-api.html).
\ No newline at end of file
+These scripts use the [Solr Collection
API](http://lucene.apache.org/solr/guide/7_4/collections-api.html).
