[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159330127 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the URL for the GeoIP database that is preloaded (and is prefilled by default) can be set to use a `file:///` location - -After installation, a custom action is available in Ambari (where stop / start services ar
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159322553 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the URL for the GeoIP database that is preloaded (and is prefilled by default) can be set to use a `file:///` location - -After installation, a custom action is available in Ambari (where stop / start services ar
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159317149 --- Diff: metron-deployment/packaging/ambari/metron-mpack/README.md --- @@ -0,0 +1,94 @@ + + +This provides a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, management and monitoring of Metron on clusters of any size. + +This allows you to easily install Metron using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + +### Prerequisites + +* Ambari 2.4.2+ + +* Installable Metron packages (either RPMs or DEBs) located in a repository on each host at `/localrepo`. + +* A [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the host running the Management and Alarm UI. + +### Quick Start + +1. Build the Metron MPack. Execute the following command from the project's root directory. +``` +mvn clean package -Pmpack -DskipTests +``` + +1. This results in the Mpack being produced at the following location. +``` + metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz +``` + +1. Copy the tarball to the host where Ambari Server is installed. + +1. Ensure that Ambari Server is stopped. + +1. Install the MPack. +``` +ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz --verbose +``` + +1. Metron swill now be available as an installable service within Ambari. + +### Installation Notes + +The MPack will make all Metron services available in Ambari in the same manner as any other services in a stack. These can be installed using Ambari's user interface using "Add Services" or during an initial cluster install. + + Co-Location + +1. The Parsers, Enrichment, Indexing, and Profiler masters should be colocated on a host with a Kafka Broker. This is necessary so that the correct Kafka topics can be created. + +1. The Enrichment and Profiler masters should be colocated on a host with an HBase client. This is necessary so that the Enrichment, Threat Intel, and Profile tables can be created. + +This colocation is currently not enforced by Ambari and should be managed by either a Service or Stack advisor as an enhancement. + + Kerberization + +The MPack allows Metron to be automatically kerberized in two different ways. +* Metron can be installed on a non-kerberized cluster and then the entire cluster can be kerberized using Ambari. +* Alternatively, Metron can be installed on top of an already kerberized cluster. + +Using the MPack is preferred, but instructions for manually Kerberizing a cluster with Metron can be found at [Kerberos-manual-setup.md](../../Kerberos-manual-setup.md). + + Zeppelin Import + +A custom action is available in Ambari to import Zeppelin dashboards. See the [metron-indexing documentation](../metron-platform/metron-indexing) for more information. + + Offline Installation + +There is only one point during installation that reaches out to the internet. That is necessary to retrieve the GeoIP database. --- End diff -- Sidenote, that should just be two slashes there (`file://`), since the extra `/` is the root dir, iirc. ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/869#discussion_r159316935
--- Diff: metron-deployment/packaging/ambari/metron-mpack/README.md ---
@@ -0,0 +1,94 @@
+
+
+This provides a Management Pack (MPack) extension for [Apache
Ambari](https://ambari.apache.org/) that simplifies the provisioning,
management and monitoring of Metron on clusters of any size.
+
+This allows you to easily install Metron using a simple, guided process.
This also allows you to monitor cluster health and even secure your cluster
with kerberos.
+
+### Prerequisites
+
+* Ambari 2.4.2+
+
+* Installable Metron packages (either RPMs or DEBs) located in a
repository on each host at `/localrepo`.
+
+* A [Node.js](https://nodejs.org/en/download/package-manager/) repository
installed on the host running the Management and Alarm UI.
+
+### Quick Start
+
+1. Build the Metron MPack. Execute the following command from the
project's root directory.
+```
+mvn clean package -Pmpack -DskipTests
+```
+
+1. This results in the Mpack being produced at the following location.
+```
+
metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz
+```
+
+1. Copy the tarball to the host where Ambari Server is installed.
+
+1. Ensure that Ambari Server is stopped.
+
+1. Install the MPack.
+```
+ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz
--verbose
+```
+
+1. Metron swill now be available as an installable service within Ambari.
+
+### Installation Notes
+
+The MPack will make all Metron services available in Ambari in the same
manner as any other services in a stack. These can be installed using Ambari's
user interface using "Add Services" or during an initial cluster install.
+
+ Co-Location
+
+1. The Parsers, Enrichment, Indexing, and Profiler masters should be
colocated on a host with a Kafka Broker. This is necessary so that the correct
Kafka topics can be created.
+
+1. The Enrichment and Profiler masters should be colocated on a host with
an HBase client. This is necessary so that the Enrichment, Threat Intel, and
Profile tables can be created.
+
+This colocation is currently not enforced by Ambari and should be managed
by either a Service or Stack advisor as an enhancement.
+
+ Kerberization
+
+The MPack allows Metron to be automatically kerberized in two different
ways.
+* Metron can be installed on a non-kerberized cluster and then the entire
cluster can be kerberized using Ambari.
+* Alternatively, Metron can be installed on top of an already kerberized
cluster.
+
+Using the MPack is preferred, but instructions for manually Kerberizing a
cluster with Metron can be found at
[Kerberos-manual-setup.md](../../Kerberos-manual-setup.md).
+
+ Zeppelin Import
+
+A custom action is available in Ambari to import Zeppelin dashboards. See
the [metron-indexing documentation](../metron-platform/metron-indexing) for
more information.
+
+ Offline Installation
+
+There is only one point during installation that reaches out to the
internet. That is necessary to retrieve the GeoIP database.
--- End diff --
Some information definitely got lost here. There's a section that was
deleted ("Installing Management Pack") that mentioned:
> Notably, the URL for the GeoIP database that is preloaded (and is
prefilled by default) can be set to use a `file:///` location
Can you modify this part here, and take a quick swing through to see if
anything else got lost during the cleanup? The geo stuff showed up in a couple
places originally, which might be why it got chopped.
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r159273549
--- Diff: pom.xml ---
@@ -159,7 +159,7 @@
${global_surefire_version}
-@{argLine} -Xmx2048m
+-Xmx2048m
--- End diff --
This was in for code coverage via JaCoCo. It basically overrides the empty
argline from above that got deleted. Was it causing problems with running
things with it there?
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159266301 --- Diff: metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.md --- @@ -441,7 +441,7 @@ Client - Kibana: * Set "kibana_es_url" to `http://:9200`. "replace_with_elasticsearch_master_hostname" is the IP of the node where you assigned ElasticSearch Master on the Assign Master tab. -* Change kibana_default_application to "dashboard/Metron-Dashboard" +* Change kibana_default_application to "dashboard/AV-YpDmwdXwc6Ua9Muh9" --- End diff -- Is it possible to provide an easily identifiable name, or are we stuck with the "AV-..." ugliness? ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159268214 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + + How do I do it? + +# Prerequisites + - A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo di
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159265938 --- Diff: metron-deployment/README.md --- @@ -1,3 +1,16 @@ +# Metron Deployment --- End diff -- As a heads up, #883 is in now, so this will have to be taken care of when you merge master to deconflict. ---
[GitHub] metron issue #885: METRON-1384 Increment master version number to 0.4.3 for ...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/885 I'm +1 by inspection, since Nick built the code base. Thanks for taking care of this! ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159246961 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + + How do I do it? + +# Prerequisites + - A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo di
[GitHub] metron issue #883: METRON-1381: Add Apache license to MD files and remove th...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/883 @nickwallen @ottobackwards I'm going to (possibly annoyingly) ping again and use the excuse that the holidays are over. ---
[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/857 @merrimanr @mmiklavc @iraghumitra What are the next steps towards being able to have this stable? It sounds like there was some playing around with versions, and that there may be some more structural changes to the code style. How do you propose we move forward? I'd really like to see this stuff made stable and I'm happy to help out with more testing or whatever else we need. ---
[GitHub] metron issue #789: METRON-1233: Remove description of Global configuration f...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/789 @DimDroll Is this still relevant with Mike's comment about #791? ---
[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/786 @merrimanr Can you deconflict this? This is otherwise ready for review, but it just needs to happen, right? ---
[GitHub] metron issue #622: METRON-1005 Create Decodable Row Key for Profiler
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/622 @nickwallen I haven't been following this discussion, but it seems like a useful feature / enhancement that's been hanging out awhile after active discussion petered out. What are the next steps here? Does this PR need changes? Should the discussion be revived on the user lists? It doesn't seem like there was any consensus on the approach, but again, I like this enhancement a lot. ---
[GitHub] metron issue #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/869 @nickwallen Thanks for adding that. I'm good with it. ---
[GitHub] metron issue #883: METRON-1381: Add Apache license to MD files and remove th...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/883 @JonZeolla Yes, I did. No idea what I was thinking, in all honesty. @nickwallen You're the actual person I wanted here. ---
[GitHub] metron pull request #785: METRON-1230: As a stopgap prior to METRON-777, add...
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/785#discussion_r159122619 --- Diff: metron-platform/metron-parsers/3rdPartyParser.md --- @@ -0,0 +1,306 @@ +# Custom Metron Parsers --- End diff -- Not sure what the state of this is, but if we're going to merge this as a stop-gap to 777, could you add the license header here? ``` ``` ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159122591 --- Diff: metron-deployment/README.md --- @@ -1,3 +1,16 @@ +# Metron Deployment --- End diff -- Can you please add the license header to this? https://github.com/apache/metron/pull/884 is close to going in and enforcing this, so I'm hoping to avoid impact to master. ``` ``` ---
[GitHub] metron issue #865: METRON-1212 The bundle System and Maven Plugin (Feature B...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/865 Given that this is a feature branch, I'm sure master will be merged in, but as a reminder, markdown file headers will be enforced by rat relatively soon (so any new markdown files in here will need it when master is merged. For reference it's: ``` ``` ---
[GitHub] metron pull request #882: METRON-1380: Create a typosquatting use-case (comm...
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/882#discussion_r159122512 --- Diff: use-cases/typosquat_detection/README.md --- @@ -0,0 +1,431 @@ +# Problem Statement --- End diff -- Can you please add the license header to this? https://github.com/apache/metron/pull/884 is close to going in and enforcing this, so I'm hoping to avoid impact to master. ``` ``` ---
[GitHub] metron pull request #884: METRON-1382 Run Stellar in a Zeppelin Notebook
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/884#discussion_r159122504 --- Diff: metron-stellar/stellar-zeppelin/README.md --- @@ -0,0 +1,85 @@ +# Stellar Interpreter for Apache Zeppelin + +[Apache Zeppelin](https://zeppelin.apache.org/) is a web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more. This project provides a means to run the Stellar REPL directly within a Zeppelin Notebook. + +## Installation --- End diff -- Can you please add the license header to this? https://github.com/apache/metron/pull/884 is close to going in and enforcing this, so I'm hoping to avoid impact to master. ``` ``` ---
[GitHub] metron issue #883: METRON-1381: Add Apache license to MD files and remove th...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/883 @ottobackwards @JonZeolla Merged in master and added header to new markdown file (the deb deployment one). Once you two are satisfied with the new commit, and I've run through the outstanding PRs for potentially problematic ones, I'm good with merging this in. ---
[GitHub] metron pull request #883: METRON-1381: Add Apache license to MD files and re...
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/883 METRON-1381: Add Apache license to MD files and remove the Rat exclusion Added an HTML comment license header to all the MD files. Removed the RAT exclusion for MD files, so we make sure to do it going forward. It can be tested by looking at the GitHub rendering, along with building the site-book and making sure no Apache headers show up. You can also create a .md file without a header and run the rat check to ensure that it catches the problem. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron METRON-1381 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/883.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #883 commit c67d3a8d3573d04628026891cc5429ed9657df5f Author: justinjleet <justinjleet@...> Date: 2017-12-23T13:07:57Z Adding all the md headers, removing rat exclusion, removing a single author listing from a file ---
[GitHub] metron issue #881: METRON-1071: Create CONTRIBUTING.md
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/881 @ottobackwards I think you're right, we should have the header in the MD files. I looked back at the discussion you kicked off, the general agreement was that we should, but we didn't follow up. I've added it here and validated that it doesn't show up in the rendering on GitHub or in the site-book. I'll be creating a Jira and PR to add it to the remaining markdown files, and removing the exclusion from the rat. @JonZeolla You okay with this change? ---
[GitHub] metron pull request #881: METRON-1071: Create CONTRIBUTING.md
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/881#discussion_r158571805 --- Diff: CONTRIBUTING.md --- @@ -0,0 +1,27 @@ +# How To Contribute +As an open source project, Metron welcomes contributions of all forms. There are several great ways to contribute! + +* [Contributing a Code Change](#contributing-a-code-change) +* Reviewing pull requests on our GitHub page. Check out current open [Pull Requests](https://github.com/apache/metron/pulls) +* Improve our documentation. Our docs are self contained in the project in README files. Doc changes is the same process as a code change. See [Contributing a Code Change](#contributing-a-code-change) +* Contributing to or starting discussions on the mailing lists. Both the user and dev lists are great places to give and receive help, or provide feedback. --- End diff -- Added the link to the site. ---
[GitHub] metron pull request #881: METRON-1071: Create CONTRIBUTING.md
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/881#discussion_r158571753 --- Diff: CONTRIBUTING.md --- @@ -0,0 +1,27 @@ +# How To Contribute +As an open source project, Metron welcomes contributions of all forms. There are several great ways to contribute! + +* [Contributing a Code Change](#contributing-a-code-change) +* Reviewing pull requests on our GitHub page. Check out current open [Pull Requests](https://github.com/apache/metron/pulls) +* Improve our documentation. Our docs are self contained in the project in README files. Doc changes is the same process as a code change. See [Contributing a Code Change](#contributing-a-code-change) --- End diff -- Good catch. Changed it. ---
[GitHub] metron pull request #881: METRON-1071: Create CONTRIBUTING.md
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/881 METRON-1071: Create CONTRIBUTING.md ## Contributor Comments Totally forgot I'd been sitting on a half done version of this. It's basically a stripped down version of the Development Guidelines to make it more obvious how to commit. Honestly, I'd rather help people out in the contributing phase than for the initial ask to be seeing if our dev guidelines can be read faster than Les Miserables. Let me know how we want to adjust this, if this is the right level of detail, etc. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron contributing Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/881.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #881 commit 676cc6a076b9858a34b659e8cad557552363029c Author: justinjleet <justinjleet@...> Date: 2017-07-29T12:04:14Z Initial contributing ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r157540768
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -33,7 +42,217 @@ For instance, an `es.date.format` of `.MM.dd.HH`
would have the consequence
roll hourly, whereas an `es.date.format` of `.MM.dd` would have the
consequence that the indices would
roll daily.
-## Using Metron with Elasticsearch 2.x
+## Upgrading to 5.6.2
+
+Users should be prepared to re-index when migrating from Elasticsearch
2.3.3 to 5.6.2. There are a number of template changes, most notably around
+string type handling, that may cause issues when upgrading.
+
+[https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html)
+
+Be aware that if you add a new string value and want to be able to filter
and search on this value from the Alerts UI, you **must** add a mapping for
that type to
+the appropriate Elasticsearch template. Below is more detail on how to
choose the appropriate mapping type for your string value.
+
+## Type Mappings
+
+Type mappings have changed quite a bit from ES 2.x -> 5.x. Here is a brief
rundown of the biggest changes. More detailed references from Elasticsearch
+are provided in the [Type Mapping References](#type-mapping-references)
section below.
+* string fields replaced by text/keyword type
+* strings have new default mappings as follows
+
+```
+{
+ "type": "text",
+ "fields": {
+"keyword": {
+ "type": "keyword",
+ "ignore_above": 256
+}
+ }
+}
+```
+
+* There is no longer a `_timestamp` field that you can set "enabled" on.
This field now causes an exception on templates.
+Replace with an application-created timestamp of "date" type.
+
+The semantics for string types have changed. In 2.x, you have the concept
of index settings as either "analyzed" or "not_analyzed" which basically means
"full text" and "keyword", respectively.
+Analyzed text basically means the indexer will split the text using a text
analyzer thus allowing you to search on substrings within the original text.
"New York" is split and indexed as two buckets,
+ "New" and "York", so you can search or query for aggregate counts for
those terms independently and will match against the individual terms "New" or
"York." "Keyword" means that the original text
+ will not be split/analyzed during indexing and instead treated as a whole
unit, i.e. "New" or "York" will not match in searches against the document
containing "New York", but searching on "New York"
+ as the full city name will. In 5.x language instead of using the "index"
setting, you now set the "type" to either "text" for full text, or "keyword"
for keywords.
+
+Below is a table depicting the changes to how String types are now handled.
+
+
+
+ sort, aggregate, or access values
+ ES 2.x
+ ES 5.x
+ Example
+
+
+ no
+
+"my_property" : {
+ "type": "string",
+ "index": "analyzed"
+}
+
+
+
+"my_property" : {
+ "type": "text"
+}
+
+Additional defaults: "index": "true", "fielddata": "false"
+
+
+ "New York" handled via in-mem search as "New" and "York"
buckets. No aggregation or sort.
+
+
+
+
+ yes
+
+
+"my_property": {
+ "type": "string",
+ "index": "analyzed"
+}
+
+
+
+"my_property": {
+ "type": "text",
+ "fielddata": "true"
+}
+
+
+
+ "New York" handled via in-mem search as "New" and "York" buckets.
Can aggregate and sort.
+
+
+
+
+ yes
+
+
+"my_property": {
+ "type": "string",
+ "index": "not_analyzed"
+}
+
+
+
+"my_property" : {
+ "type": "keyword"
+}
+
+
+
+ "New York" searchable as single value. Can aggregate
and sort. A search for "New" or "Yo
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r157540388 --- Diff: metron-platform/metron-elasticsearch/README.md --- @@ -1,5 +1,14 @@ # Elasticsearch in Metron +## Table of Contents + +* [Introduction](#introduction) +* [Properties](#properties) +* [Upgrading to 5.6.2](#upgrading-to-562) +* [Type Mappings](#type-mappings) +* [Using Metron with Elasticsearch 5.x](#using-metron-with-elasticsearch-5x) --- End diff -- This link is broken, because the section name doesn't line up with the actual name ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r157238284 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,127 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + + How do I do it? + +# Prerequisites + - A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo di
[GitHub] metron issue #867: METRON-1350: Add reservoir sampling functions to Stellar
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/867 @ottobackwards Left a comment on the ticket you made: https://issues.apache.org/jira/browse/METRON-1361?focusedCommentId=16291159=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16291159 ---
[GitHub] metron issue #866: METRON-1349 Full Dev Builds Metron Twice
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/866 I'm +1 by inspection, given that @ottobackwards ran it up successfully. Good job! ---
[GitHub] metron issue #866: METRON-1349 Full Dev Builds Metron Twice
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/866 @ottobackwards Nope, at least I don't. I assume you're not on a Mac, because I don't believe cowsay is installed by default on them. Hilariously, I just installed it via homebrew though. ---
[GitHub] metron issue #866: METRON-1349 Full Dev Builds Metron Twice
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/866 Yeah, sorry, I wasn't clear, my log sample is prior to this PR ---
[GitHub] metron issue #866: METRON-1349 Full Dev Builds Metron Twice
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/866 @ottobackwards Yes, I think you'd see it twice, looking at my logs ``` 2017-12-11 09:50:06,120 p=15256 u=jleet | TASK [metron-builder : Build Metron] *** 2017-12-11 09:56:16,676 p=15256 u=jleet | [0;33mchanged: [node1 -> localhost][0m 2017-12-11 09:56:16,682 p=15256 u=jleet | TASK [metron-builder : Build Metron RPMs] ** 2017-12-11 10:07:05,114 p=15256 u=jleet | [0;33mchanged: [node1 -> localhost][0m ``` Way down later in the build: ``` 2017-12-11 10:36:16,385 p=15256 u=jleet | TASK [metron-builder : Build Metron] *** 2017-12-11 10:46:43,988 p=15256 u=jleet | [0;33mchanged: [node1 -> localhost][0m 2017-12-11 10:46:44,014 p=15256 u=jleet | TASK [metron-builder : Build Metron RPMs] ** 2017-12-11 11:00:49,602 p=15256 u=jleet | [0;33mchanged: [node1 -> localhost][0m ... ``` @nickwallen Is this a correct assessment? ---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/858 @merrimanr Is it worth having a new discuss thread where you lay out what you've done, where you expect this to end, and what (if any) work that would be nice to have but isn't essential for this to be feature complete? i.e. is the endpoint where we merge to master (and I'm injecting my own bias here, so feel free to adjust as you see fit): 1. All e2e and int tests run on common infrastructure. 1. All e2e and int tests are run automatically in the Travis build. 1. All e2e and int tests run repeatably and reliably in the Travis build. 1. Debugging options are available and documented. 1. The new infra and how to interact with it is documented. 1. Old infrastructure removed (anything unused or commented out is deleted, instead of staying). @ottobackwards @cestella Is that reasonable? I don't really want to make decisions regarding the specific end criteria of a feature branch in the POC PR. Sidenote, I like the top level PR and the child tasks for this. It'd be a good way to both organize and flesh out the TODO list you had and give everyone an easy way to see where things are (and where they can hop in and help out) ---
[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/857 I've hit the same intermittent errors trying this, e.g. ``` [INFO] - Failed: unknown error: Element ... is not clickable at point (1279, 95). Other element would receive the click: ... ``` One of the very first things that failed was ``` [INFO] ✗ should display error message for invalid credentials [INFO] - Failed: Error while waiting for Protractor to sync with the page: true ``` Is it possible there's an intermittent error getting the tests initially lined up that just carries through everything? @iraghumitra Can you also update the README with the instructions for running this with the mvn profile + any troubleshooting or other steps that need to happen? ---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/858 I think getting the feature branch set up was a great first step, so thanks for setting that up, @merrimanr. I agree with @ottobackwards, that we need a discussion on what the next steps are. For me, I think it's getting reqs fleshed out and agreed on (i.e. what's the end state where we pull in the branch, and what's follow on). I think getting a clear idea of end state for this phase is going to help avoid feature creep and the branch being stagnant. ---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/858 I agree with @cestella (and this might spill over into a discuss thread as @ottobackwards mentioned). Maintaining both anything over other than short, short term is going to be a nightmare. Invariably one or the other is going to break independently of the other. Assuming we do go with remote debugging of Docker (and that's a total assumption, to be clear), we should make sure that the instructions aren't IntelliJ dependent. Or at least, there are best effort instructions for people using something else. I.e. if we end up using something like JetBrains' Docker Integration plugin, I personally think we should have at least some debugging instructions available outside that. ---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/858 @merrimanr Did you look into what migrating our non-e2e tests integration tests would involve? I think for a POC, it's important to get a sense of how we'd be able to unify the infrastructure of these, even if they aren't necessarily all unified at POC time. If not, are you planning on investigating that once you resolve the current list of e2e issues? ---
[GitHub] metron issue #854: Experimental Improvements - Feedback Only - Do Not Merge
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/854 Casey and I investigated this previously during one of the times we were running into the upper time limit. The main reason we ultimately chose not to do this was because we share the build queue with other Apache projects and kicking up multiple builds (at a period in time was backing up in the queues) seemed like it could cause problems for other projects. ---
[GitHub] metron issue #823: METRON-1286 Add MIN & MAX Stellar functions
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/823 +1, I'm set ---
[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/857 I haven't looked at the code yet, but I was able to pull this down and run the e2e tests repeatedly without failures, so this is definitely great stuff. ---
[GitHub] metron pull request #823: METRON-1286 Add MIN & MAX Stellar functions
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/823#discussion_r153919356
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/OrdinalFunctions.java
---
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class OrdinalFunctions {
+
+/**
+ * Stellar Function: MAX
+ *
+ * Return the maximum value of a list of input values in a Stellar list
+ */
+@Stellar(name = "MAX"
+, description = "Returns the maximum value of a list of input
values"
+, params = {"list_of_values - Stellar list of values to
evaluate. The list may only contain 1 type of object (only strings or only
numbers)" +
+" and the objects must be comparable / ordinal"}
+, returns = "The highest value in the list, null if the list
is empty or the input values could not be ordered")
+public static class Max extends BaseStellarFunction {
+
+@Override
+public Object apply(List args) {
+if (args.size() < 1 || args.get(0) == null) {
+throw new IllegalStateException("MAX function requires at
least a Stellar list of values");
+}
+List list = (List) args.get(0);
+return orderList(list, true);
+}
+}
+
+/**
+ * Stellar Function: MIN
+ *
+ * Return the minimum value of a list of input values in a Stellar list
+ */
+@Stellar(name = "MIN"
+, description = "Returns the minimum value of a list of input
values"
+, params = {"list_of_values - Stellar list of values to
evaluate. The list may only contain 1 type of object (only strings or only
numbers)" +
+" and the objects must be comparable / ordinal"}
+, returns = "The lowest value in the list, null if the list is
empty or the input values could not be ordered")
+public static class Min extends BaseStellarFunction {
+@Override
+public Object apply(List args) {
+if (args.size() < 1 || args.get(0) == null) {
+throw new IllegalStateException("MIN function requires at
least a Stellar list of values");
+}
+List list = (List) args.get(0);
+return orderList(list, false);
+}
+}
+
+private static Object orderList(List list, Boolean max) {
+if (list.isEmpty()) {
+return null;
+}
+List filteredList = (List) list.stream().filter(index ->
!(index == null)).collect(Collectors.toList());
+if (filteredList.isEmpty()) {
+return null;
+}
+try {
+if (max) {
+Collections.sort(filteredList,Collections.reverseOrder());
--- End diff --
Well, that is disappointing, but makes sense.
---
[GitHub] metron pull request #823: METRON-1286 Add MIN & MAX Stellar functions
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/823#discussion_r153899916
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/OrdinalFunctions.java
---
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class OrdinalFunctions {
+
+/**
+ * Stellar Function: MAX
+ *
+ * Return the maximum value of a list of input values in a Stellar list
+ */
+@Stellar(name = "MAX"
+, description = "Returns the maximum value of a list of input
values"
+, params = {"list_of_values - Stellar list of values to
evaluate. The list may only contain 1 type of object (only strings or only
numbers)" +
+" and the objects must be comparable / ordinal"}
+, returns = "The highest value in the list, null if the list
is empty or the input values could not be ordered")
+public static class Max extends BaseStellarFunction {
+
+@Override
+public Object apply(List args) {
+if (args.size() < 1 || args.get(0) == null) {
+throw new IllegalStateException("MAX function requires at
least a Stellar list of values");
+}
+List list = (List) args.get(0);
+return orderList(list, true);
+}
+}
+
+/**
+ * Stellar Function: MIN
+ *
+ * Return the minimum value of a list of input values in a Stellar list
+ */
+@Stellar(name = "MIN"
+, description = "Returns the minimum value of a list of input
values"
+, params = {"list_of_values - Stellar list of values to
evaluate. The list may only contain 1 type of object (only strings or only
numbers)" +
+" and the objects must be comparable / ordinal"}
+, returns = "The lowest value in the list, null if the list is
empty or the input values could not be ordered")
+public static class Min extends BaseStellarFunction {
+@Override
+public Object apply(List args) {
+if (args.size() < 1 || args.get(0) == null) {
+throw new IllegalStateException("MIN function requires at
least a Stellar list of values");
+}
+List list = (List) args.get(0);
+return orderList(list, false);
+}
+}
+
+private static Object orderList(List list, Boolean max) {
+if (list.isEmpty()) {
+return null;
+}
+List filteredList = (List) list.stream().filter(index ->
!(index == null)).collect(Collectors.toList());
+if (filteredList.isEmpty()) {
+return null;
+}
+try {
+if (max) {
+Collections.sort(filteredList,Collections.reverseOrder());
--- End diff --
Why not just use Collections.min() and Collections.max()?
---
[GitHub] metron issue #852: METRON-1239 Drop extra dev environments
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/852 Looks like @merrimanr commented on the email thread, so I'll copy it here for posterity. > I wrote the ReadMeUtils class a long time ago as a way to make documenting the REST endpoints easier. The Controller class methods are annotated so that endpoint documentation is displayed in Swagger but it is also duplicated in the README. It seemed like a good idea at the time to provide a utility to make this easier so that you only had to document in one place. It was actually helpful (to me anyways) when we first introduced a large number of REST endpoints and saved some tedious copy/pasting. > > In hindsight, there was no way of enforcing that we use the utility along with the `README.vm` template. People intuitively edit the README.md instead and the template quickly became stale. Eventually I got tired of keeping the template in sync so I stopped using it as well. This class can (and should) be safely removed. I'd say just dump the template and the utility in this PR, since you'd already either have to clean it up or wait for another PR anyway. ---
[GitHub] metron issue #852: METRON-1239 Drop extra dev environments
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/852 Glancing briefly, it looks like `ReadMeUtils` uses it as a template for the metron-rest README.md. Just running the main in there overwrites the metron-rest README.md. Which seems very odd, given that `ReadMeUtils` is in the test package. There seems to be no documentation of this class, or its purpose, and I didn't dig enough into the code to figure it out. Even not knowing the details and assuming I'm not misreading what's happening, I don't like that there's an expectation of editing a `README.vm` file, then running a program to produce the final output `README.md`. `README.md` can vary independently of `README.vm`. And it already has. It's outside the scope of this ticket, but at minimum, that class needs to be moved out of test, it needs to be actually documented what the purpose of it is, the steps to use it, etc. Right now, though, unless someone comes up with a compelling reason not to, I'm in favor of killing it entirely. I don't ever see that being properly managed, even if it does have some utility built in. ---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/803 I agree. I'm fine with going ahead with this, but I'd like to see end to end stability being addressed as the next UI priority, which I believe @iraghumitra is already doing some work on. +1 ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r153289006
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/snort_index.template
---
@@ -102,13 +94,25 @@
"match_mapping_type": "*"
}
},
- {
-"threat_triage_reason": {
- "mapping": {
-"type": "string"
- },
- "match": "threat:triage:rules:*:reason",
- "match_mapping_type": "*"
+{
+ "threat_triage_reason": {
+"mapping": {
+ "type": "text",
+ "fielddata": "true"
+},
+"match": "threat.triage.rules:*:reason",
+"match_mapping_type": "*"
+ }
+},
+{
+ "threat_triage_name": {
+"mapping": {
+ "type": "text",
+ "fielddata": "true"
+},
+"match": "threat.triage.rules:*:name",
+"match_mapping_type": "*"
+ }
}
},
--- End diff --
This brace is extraneous, I'm guessing a merge broke it. Drop it, but keep
the comma and we should be good. I'd just get the formatting lined back up
while you're in there.
---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/803 @nickwallen I agree, unless I'm missing something, it seems like a bug with recent searches, since you didn't say searches were weird during the initial attempts. ---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/803 @nickwallen I haven't been able to get into the same sort of state, but I know you've used/tested the UI more than me. Do you know a repeatable way to reproduce this? Or have you been able to determine if it's preexisting, assuming you've looked into it. ---
[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/803#discussion_r152310478
--- Diff:
metron-interface/metron-alerts/e2e/alerts-list/meta-alerts/meta-alert.po.ts ---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {browser, element, by} from 'protractor';
+
+export class MetaAlertPage {
+
+ getPageTitle() {
+return element(by.css('app-meta-alerts .form-title')).getText();
+ }
+
+ getMetaAlertsTitle() {
+return element(by.css('app-meta-alerts .title')).getText();
+ }
+
+ getAvailableMetaAlerts() {
+return element(by.css('app-meta-alerts .guid-name-container
div')).getText();
+ }
+
+ selectRadio() {
+return element.all(by.css('app-meta-alerts .checkmark')).click();
+ }
+
+ addToMetaAlert() {
+
element.all(by.css('app-meta-alerts')).get(0).element(by.buttonText('ADD')).click();
+browser.sleep(2000);
--- End diff --
@iraghumitra any thoughts on this?
---
[GitHub] metron issue #845: METRON-1321 Metaalert Threat Score Type Does Not Match Se...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/845 +1 by inspection, assuming @ottobackwards is good. Thanks for expanding the comments out, it's definitely helpful. ---
[GitHub] metron issue #845: METRON-1321 Metaalert Threat Score Type Does Not Match Se...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/845 Could we also add the threat score to the metaalert template, to match the other templates? ---
[GitHub] metron pull request #845: METRON-1321 Metaalert Threat Score Type Does Not M...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/845#discussion_r152075990
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
---
@@ -614,8 +625,15 @@ protected void calculateMetaScores(Document metaAlert)
{
}
metaScores = new MetaScores(scores);
}
+
+// add a summary (max, min, avg, count, sum) of all the threat scores
from the child alerts
metaAlert.getDocument().putAll(metaScores.getMetaScores());
-metaAlert.getDocument().put(threatTriageField,
metaScores.getMetaScores().get(threatSort));
+
+// the overall threat score for the metaalert; either max, min, avg,
count or sum of all child scores
--- End diff --
The calculations were done as Double and given to ES. However, there's no
definition of the field in ES (It just used automatic mapping), so it was given
the ES double.
---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/803
@iraghumitra looks like the new API isn't being used quite right.
Sample from the dev tools
```
{
"alerts": [
{
"guid": "50a0c1f6-8a55-4cdd-a031-81c53174ad7b",
"sensorType": [
"snort"
],
"index": "snort_index_2017.11.20.15"
},
...
```
This should be, I believe,
```
{
"alerts": [
{
"guid": "50a0c1f6-8a55-4cdd-a031-81c53174ad7b",
"sensorType": "snort",
"index": "snort_index_2017.11.20.15"
},
...
],
"groups": [
"source:type"
]
}
```
---
[GitHub] metron issue #832: METRON-1301 Sorting on Triage Score Unexpectedly Filters ...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/832 Spun this up on full dev, and saw that sorting worked as expected in the UI. This worked both when no groupings were selected and also when drilling down (e.g. grouping by ip_dst_addr and country). I also like the refactorings a lot. It's good stuff that makes life easier and more testable. +1, assuming @cestella is happy with the changes you made on his comments. ---
[GitHub] metron issue #827: METRON-1294: IP addresses are not formatted correctly in ...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/827 Great, thanks for the update. +1 ---
[GitHub] metron issue #827: METRON-1294: IP addresses are not formatted correctly in ...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/827 @nickwallen The parameter name is preexisting and I'm fine with leaving that. Would you be okay with changing the descriptions in the README and annotations? That should be a nonintrusive change that doesn't require spinning everything back up. @merrimanr Do you have any objections to doing that as a middle ground? ---
[GitHub] metron issue #827: METRON-1294: IP addresses are not formatted correctly in ...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/827
I ran a request giving sensors:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '["snort", "bro"]'
'http://node1:8082/api/v1/search/column/metadata'
```
which returns fine
```
{
"TTLs": "double",
"bro_timestamp": "string",
"enrichments:geo:ip_dst_addr:location_point": "other",
"sha256": "string",
"enrichmentjoinbolt:joiner:ts": "date",
"certificate:version": "integer",
...
}
```
but giving the actual indices returns nothing, e.g.
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '["bro_index_2017.11.17.14"]'
'http://node1:8082/api/v1/search/column/metadata'
```
I assume it's intentional that indices don't actually return data, which
I'm fine with, but we need to rename things from indices to sensor or something.
---
[GitHub] metron issue #827: METRON-1294: IP addresses are not formatted correctly in ...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/827 I tried hitting the `/api/v1/search/column/metadata` endpoint on fulldev with `["madeupindex"]`; e.g. curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '["madeupindex"]' 'http://node1:8082/api/v1/search/column/metadata' I would expect this to return no results, because the index doesn't exist, but instead I get back a lot of fields. ---
[GitHub] metron issue #825: METRON-1290: Only first 10 alerts are update when a MetaA...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/825 @nickwallen https://github.com/apache/metron/pull/842 @merrimanr If you're good with that PR, feel free to close this one. ---
[GitHub] metron pull request #842: METRON-1290: Only first 10 alerts are update when ...
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/842 METRON-1290: Only first 10 alerts are update when a MetaAlert status is changed to inactive ## Contributor Comments This PR supercedes https://github.com/apache/metron/pull/825. The fix was already in place as a result of refactoring in https://github.com/apache/metron/pull/824. This just updates the status integration test to use above the threshold of 10. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron METRON-1290 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/842.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #842 commit 9548ab44e1e4c0cdd4e7c3c91f62037bfcf54f70 Author: justinjleet <justinjl...@gmail.com> Date: 2017-11-16T15:33:56Z Refactored metaalert status test to be above default returned results ---
[GitHub] metron issue #825: METRON-1290: Only first 10 alerts are update when a MetaA...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/825 @nickwallen I'm about to open a PR with an updated integration test. We should be able to close this one, since the changes here aren't relevant anymore. ---
[GitHub] metron issue #825: METRON-1290: Only first 10 alerts are update when a MetaA...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/825 @merrimanr This is already taken care of by the various refactoring in https://github.com/apache/metron/pull/824, right? Can you close this (and the associated jira) if that's accurate? ---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/824 +1, looks good. Thanks for all the work on the supplemental fixes. Feel free to skip attribution on the testing PR. ---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/824 Code-wise, I'm pretty good at this point. Once the docs come in, I'll give them a once-over and hopefully we're good to go soon. Thanks a lot for the hard work here! ---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/824
## Patch neither alert and status
Create a metaalerts and get the GUID for the following steps.
### Patch in new field
```
/api/v1/update/patch
curl -X PATCH --header 'Content-Type: application/json' --header 'Accept:
*/*' -d '{
"guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
"index": "metaalert_index",
"patch": [
{
"op": "add"
, "path": "/name"
, "value": "My new meta alert name"
}
],
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/update/patch'
```
### Retrieve the meta alert and ensure it contains the new 'name' field
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
"index": "metaalert_index",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/824
## Patch alert and status
Create a metaalerts and get the GUID for the following steps.
### Attempt to update status field
```
/api/v1/update/patch
curl -X PATCH --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
"index": "metaalert_index",
"patch": [
{
"op": "replace"
, "path": "/status"
, "value": "failure"
}
],
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/update/patch'
```
Should return
```
{
"responseCode": 500,
"message": "Meta alert patches are not allowed for /alert or /status
paths. Please use the add/remove alert or update status functions instead.",
"fullMessage": "IllegalArgumentException: Meta alert patches are not
allowed for /alert or /status paths. Please use the add/remove alert or update
status functions instead."
}
```
### Attempt to update the alert list
```
/api/v1/update/patch
curl -X PATCH --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
"index": "metaalert_index",
"patch": [
{
"op": "replace"
, "path": "/alert"
, "value": [{
"alertOne":"test"
},
{
"alertTwo":"test"
}
]
}
],
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/update/patch'
```
Should return
```
{
"responseCode": 500,
"message": "Meta alert patches are not allowed for /alert or /status
paths. Please use the add/remove alert or update status functions instead.",
"fullMessage": "IllegalArgumentException: Meta alert patches are not
allowed for /alert or /status paths. Please use the add/remove alert or update
status functions instead."
}
```
---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/824
## Create meta alert with more than 10 alerts
### Find more than 10 alerts alerts
```
/api/v1/search/search
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"fields": [
"guid"
],
"from": 0,
"indices": [
"snort"
],
"query": "ip_dst_addr:192.168.66.121",
"size": 11
}' 'http://node1:8082/api/v1/search/search'
```
Note the alerts that come back
```
62a53a5f-78e6-417a-8078-fb850baa3e84
876b72cb-9d72-4706-ac99-46cf91a8f359
5fd8b0a0-1f68-494a-ae20-633542a7045d
aee597a0-4255-499a-a4e2-ec7d756babb2
bf9e0e73-e64c-4759-b4f7-efad0a60be82
5ab9ce98-30db-45b2-a4e6-6489f136c839
0a4a7019-04f8-4a8c-af0b-d2e3908ecdc9
3423fdca-cefa-402a-b57d-60b75a15f046
2eb63002-e5f2-467a-8675-30b653ae145b
53f38cfd-aa89-4e49-ba5f-827eb73774cd
5f71a515-4976-4b0d-be85-bb6879b1e151
```
### Create a metaalert with the alerts
```
/api/v1/metaalert/create
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"alerts": [
{
"guid": "62a53a5f-78e6-417a-8078-fb850baa3e84",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid":"876b72cb-9d72-4706-ac99-46cf91a8f359",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid": "5fd8b0a0-1f68-494a-ae20-633542a7045d",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid": "aee597a0-4255-499a-a4e2-ec7d756babb2",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid":"bf9e0e73-e64c-4759-b4f7-efad0a60be82",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid": "5ab9ce98-30db-45b2-a4e6-6489f136c839",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid":"0a4a7019-04f8-4a8c-af0b-d2e3908ecdc9",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid": "3423fdca-cefa-402a-b57d-60b75a15f046",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid": "2eb63002-e5f2-467a-8675-30b653ae145b",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid":"53f38cfd-aa89-4e49-ba5f-827eb73774cd",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid":"5f71a515-4976-4b0d-be85-bb6879b1e151",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
}
],
"groups": [
"test"
]
}' 'http://node1:8082/api/v1/metaalert/create'
```
Make sure to get the resulting guid from the response.
```
00eae5ba-6137-4601-ae3a-fbf0003e58e6
```
### Retrieve the meta alert and ensure it contains the provided alerts
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
"index": "metaalert_index",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
### Retrieve the child alerts
Ensure all alerts have the 'metaalerts' field populated with the parent
meta alert.
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid":"62a53a5f-78e6-417a-8078-fb850baa3e84",
"sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
... // 10 more times
```
---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/824
## Changing Metaalert status
### Find two alerts
```
/api/v1/search/search
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"fields": [
"guid"
],
"from": 0,
"indices": [
"snort"
],
"query": "ip_dst_addr:192.168.66.121",
"size": 2
}' 'http://node1:8082/api/v1/search/search'
```
Results in two guids:
```
8b8314d4-277b-44dc-a75b-04b0cdcedb40
4ac26cf7-ab93-4940-9a0e-8e7f4d67736d
```
### Create a metaalert with the alerts
```
/api/v1/metaalert/create
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"alerts": [
{
"guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
}
],
"groups": [
"test"
]
}' 'http://node1:8082/api/v1/metaalert/create'
```
Make sure to get the resulting guid from the response.
```
da60ccc9-9e79-45c5-be07-0a322c8791f0
```
### Retrieve the meta alert and ensure it contains the provided alerts
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "da60ccc9-9e79-45c5-be07-0a322c8791f0",
"index": "metaalert_index",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
### Change the meta alert status to active
This makes sure nothing happens when we set active status to the same
active status.
```
/api/v1/metaalert/update/status/{guid}/{status}
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json'
'http://node1:8082/api/v1/metaalert/update/status/da60ccc9-9e79-45c5-be07-0a322c8791f0/active'
```
It should return false, as no status has changed.
### Retrieve the metaalert and ensure it is still active
Look for the 'status' field.
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "da60ccc9-9e79-45c5-be07-0a322c8791f0",
"index": "metaalert_index",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
### Retrieve the child alerts
Ensure both alerts have the 'metaalerts' field populated with the parent
meta alert.
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
"sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
"sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
```
### Change the meta alert status to inactive
Look for the 'status' field. Running this once will set it to 'inactive'.
Subsequent runs have no effect ('inactive' -> 'inactive' does nothing).
```
/api/v1/metaalert/update/status/{guid}/{status}
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json'
'http://node1:8082/api/v1/metaalert/update/status/da60ccc9-9e79-45c5-be07-0a322c8791f0/inactive'
```
It should return true, because the status has changed.
### Retrieve the metaalert and ensure it is inactive
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "da60ccc9-9e79-45c5-be07-0a322c8791f0",
"index": "metaalert_index",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
### Retrieve the child alerts
Ensure neither alert has the 'metaalerts' field populated with the parent
meta alert.
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/js
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/824
## Removing alerts and removing an already removed alert
### Find two alerts
```
/api/v1/search/search
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"fields": [
"guid"
],
"from": 0,
"indices": [
"snort"
],
"query": "ip_dst_addr:192.168.66.121",
"size": 2
}' 'http://node1:8082/api/v1/search/search'
```
Results in two guids:
```
8b8314d4-277b-44dc-a75b-04b0cdcedb40
4ac26cf7-ab93-4940-9a0e-8e7f4d67736d
```
### Create a metaalert with the alerts
```
/api/v1/metaalert/create
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"alerts": [
{
"guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
}
],
"groups": [
"test"
]
}' 'http://node1:8082/api/v1/metaalert/create'
```
Make sure to get the resulting guid from the response.
```
b25b663e-39c9-42d5-a52c-e6380235d43f
```
### Retrieve the meta alert and ensure it contains the provided alerts
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "b25b663e-39c9-42d5-a52c-e6380235d43f",
"index": "metaalert_index",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
### Remove one of the alerts
```
/api/v1/metaalert/remove/alert
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"alerts": [
{
"guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
}
],
"metaAlertGuid": "b25b663e-39c9-42d5-a52c-e6380235d43f"
}' 'http://node1:8082/api/v1/metaalert/remove/alert'
```
### Retrieve the meta alert again, and ensure it only contains the second
alert.
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "b25b663e-39c9-42d5-a52c-e6380235d43f",
"index": "metaalert_index",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
### Rerun the delete
```
/api/v1/metaalert/remove/alert
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"alerts": [
{
"guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
}
],
"metaAlertGuid": "b25b663e-39c9-42d5-a52c-e6380235d43f"
}' 'http://node1:8082/api/v1/metaalert/remove/alert'
```
### Retrieve the meta alert again, and ensure it only contains the second
alert.
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "b25b663e-39c9-42d5-a52c-e6380235d43f",
"index": "metaalert_index",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
### Retrieve the child alerts
Ensure only the second alert has the 'metaalerts' field populated with the
parent met alert.
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
"sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
"sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
```
---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/824
## Adding alerts and adding a preexisting alert
### Find two alerts
```
/api/v1/search/search
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"fields": [
"guid"
],
"from": 0,
"indices": [
"snort"
],
"query": "ip_dst_addr:192.168.66.121",
"size": 2
}' 'http://node1:8082/api/v1/search/search'
```
Results in two guids:
```
8b8314d4-277b-44dc-a75b-04b0cdcedb40
4ac26cf7-ab93-4940-9a0e-8e7f4d67736d
```
### Create a metaalert with only one of the alerts
```
/api/v1/metaalert/create
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"alerts": [
{
"guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
}
],
"groups": [
"test"
]
}' 'http://node1:8082/api/v1/metaalert/create'
```
Make sure to get the resulting guid from the response.
```
6a4affe4-02ce-4d25-80b1-bfc4ca53f557
```
### Retrieve the meta alert and ensure it contains the provided alert
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "6a4affe4-02ce-4d25-80b1-bfc4ca53f557",
"index": "metaalert_index",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
### Retrieve the child alert and ensure the 'metaalerts' field contains the
new GUID of the new metaalert
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
"sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
```
### Add the same alert to the meta alert
```
/api/v1/metaalert/add/alert
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"alerts": [
{
"guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
}
],
"metaAlertGuid": "6a4affe4-02ce-4d25-80b1-bfc4ca53f557"
}' 'http://node1:8082/api/v1/metaalert/add/alert'
```
It should return "false" as no alerts have been added. The meta alert
should be retrieved again to validate.
### Run the add alert again but with the second alert
```
/api/v1/metaalert/add/alert
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"alerts": [
{
"guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
},
{
"guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
"index": "snort_index_2017.11.15.17",
"sensorType": "snort"
}
],
"metaAlertGuid": "6a4affe4-02ce-4d25-80b1-bfc4ca53f557"
}' 'http://node1:8082/api/v1/metaalert/add/alert'
```
It should return true, because the second alert will be added. The meta
alert should be retrieved again to validate.
### Retrieve the child alerts
Ensure they have the 'metaalerts' field populated with their parent.
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
"sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
"sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
```
---
[GitHub] metron pull request #824: METRON-1289: Alert fields are lost when a MetaAler...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/824#discussion_r151203066
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/MetaAlertController.java
---
@@ -60,5 +63,37 @@
) throws RestException {
return new ResponseEntity<>(metaAlertService.create(createRequest),
HttpStatus.OK);
}
+
+ @ApiOperation(value = "Create a meta alert")
--- End diff --
The descriptions and so on need to be updated on all of these. I know
you're working on documentation, but I wanted to call it out so it doesn't slip
through.
---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/824 Double check me on that logic though. I could definitely be masking an off by one error there. ---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/824 @merrimanr I'm okay with excluding metaalerts (although I need to review what you did there). I wouldn't expect it to go down by two though. Say I have two matches, I put one in a metaalert (so it should be hidden). I make the query again. I would still expect to get the remaining, standalone match. The metaalert should never have showed up and the child alert should be hidden. So one result. ---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/824
@iraghumitra I'm unable to duplicate the grouping on the current code
(although I admittedly ran through a pretty basic example). Here's what I did,
so let me know if I missed something, or you have a particular query you made.
### Find an alert
```
/api/v1/search/search
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"fields": [
"guid"
],
"from": 0,
"indices": [
"snort"
],
"query": "ip_dst_addr:192.168.66.121",
"size": 1
}' 'http://node1:8082/api/v1/search/search'
```
In this case, I just grabbed one:
```
"d29925a7-da80-49d0-b7b5-0663380d526f":"snort_index_2017.11.14.15"
```
### Retrieve that alert from findOne to get something to group by
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "d29925a7-da80-49d0-b7b5-0663380d526f",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
I'll group, arbitrarily, with:
```
"ip_dst_addr": "192.168.66.121"
```
### Run a group by
```
/api/v1/search/group
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"groups": [
{
"field": "ip_dst_addr"
}
],
"indices": [
"snort",
"metaalert"
],
"query":"ip_dst_addr:192.168.66.121"
}' 'http://node1:8082/api/v1/search/group'
```
Results in:
```
{
"groupedBy": "ip_dst_addr",
"groupResults": [
{
"key": "192.168.66.121",
"total": 1299,
"score": null
}
]
}
```
### Create a metaalert with that GUID
```
/api/v1/metaalert/create
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"groups": [
"test"
],
"guidToIndices": {
"adf65dbb-73fd-4347-a61d-a990166fbbb1":"snort_index_2017.11.14.15"
}
}' 'http://node1:8082/api/v1/metaalert/create'
```
It returns the guid, in this case:
```
715538d9-7f4f-4316-b149-a9659df06de4
```
### (Optionally) Look up the metaalert to double check it was created as
expected.
```
/api/v1/search/findOne
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "715538d9-7f4f-4316-b149-a9659df06de4",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
### Run the group by from before
```
/api/v1/search/group
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"groups": [
{
"field": "ip_dst_addr"
}
],
"indices": [
"snort",
"metaalert"
],
"query":"ip_dst_addr:192.168.66.121"
}' 'http://node1:8082/api/v1/search/group'
```
Validate that the count has gone down by one (since the metaalert doesn't
return and the child alert is now hidden:
```
{
"groupedBy": "ip_dst_addr",
"groupResults": [
{
"key": "3232252537",
"total": 1298,
"score": null
}
]
}
```
---
[GitHub] metron pull request #824: METRON-1289: Alert fields are lost when a MetaAler...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/824#discussion_r150892216
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
---
@@ -135,8 +138,9 @@ private Document getDocumentFromResult(Result result)
throws IOException {
Map.Entry<byte[], byte[]> entry= columns.lastEntry();
Long ts = Bytes.toLong(entry.getKey());
if(entry.getValue()!= null) {
- String json = new String(entry.getValue());
- return new Document(json, Bytes.toString(result.getRow()), null, ts);
+ Map<String, Object> json = JSONUtils.INSTANCE.load(new
String(entry.getValue()), new TypeReference<Map<String, Object>>() {
+ });
+ return new Document(json, Bytes.toString(result.getRow()), (String)
json.get(SOURCE_TYPE), ts);
--- End diff --
Yeah, let's go with this. It solves several problems nicely
---
[GitHub] metron pull request #824: METRON-1289: Alert fields are lost when a MetaAler...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/824#discussion_r150872247
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
---
@@ -135,8 +138,9 @@ private Document getDocumentFromResult(Result result)
throws IOException {
Map.Entry<byte[], byte[]> entry= columns.lastEntry();
Long ts = Bytes.toLong(entry.getKey());
if(entry.getValue()!= null) {
- String json = new String(entry.getValue());
- return new Document(json, Bytes.toString(result.getRow()), null, ts);
+ Map<String, Object> json = JSONUtils.INSTANCE.load(new
String(entry.getValue()), new TypeReference<Map<String, Object>>() {
+ });
+ return new Document(json, Bytes.toString(result.getRow()), (String)
json.get(SOURCE_TYPE), ts);
--- End diff --
I agree with @cestella, I'd rather use the existing abstraction. Setting a
field transformer is a cleaner way of doing things, in my opinion, than
requiring the user to know know and input where all the GUIDs are coming from.
Another concern about ES5 is that PR may live for awhile, even after we're
fairly confident in it (there's going to be a lot of testing involved there).
Casey's PR is a pretty simple way of handling it (set a config, pass it in, use
it basically the same way you set it up).
---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/824 @iraghumitra Do you have the specific metaalerts and calls that you made? I'm spinning this up again, but it'll be a bit before I can test something myself. ---
[GitHub] metron pull request #824: METRON-1289: Alert fields are lost when a MetaAler...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/824#discussion_r150854643
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
---
@@ -135,8 +138,9 @@ private Document getDocumentFromResult(Result result)
throws IOException {
Map.Entry<byte[], byte[]> entry= columns.lastEntry();
Long ts = Bytes.toLong(entry.getKey());
if(entry.getValue()!= null) {
- String json = new String(entry.getValue());
- return new Document(json, Bytes.toString(result.getRow()), null, ts);
+ Map<String, Object> json = JSONUtils.INSTANCE.load(new
String(entry.getValue()), new TypeReference<Map<String, Object>>() {
+ });
+ return new Document(json, Bytes.toString(result.getRow()), (String)
json.get(SOURCE_TYPE), ts);
--- End diff --
I would prefer to see one of two things happen here. Either we keep the
constant in the ES specific classes (which is admittedly less than ideal, but
it does limit the pollution of ES knowledge into HBase classes) and populate
source type from there (basically moving the loading and source type population
there). Alternatively, we pass in a more general function that can be applied
to the fields and configure and handle it appropriately.
I think the second one is probably more general useful to be able to do,
but given the state of ES5 upgrade making this particular case obsolete, I'm
amenable to doing the first option.
At bare minimum we should replace the '.'s with ':'s only if present. Even
if there's not a Solr implementation, I don't want HBaseDao tied to ES so
directly.
@cestella Do you have a preference on implementation? I know you'd had
some comments earlier, but I don't want to put words in your mouth.
---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/824
I've spun this up, and `add/alert` throws this exception:
```
{
"responseCode": 500,
"message": "class org.apache.metron.elasticsearch.dao.ElasticsearchDao:
ElasticsearchDao upsert failed: failure in bulk execution:
[1]: index [metaalert_index], type [null_doc], id
[be44748e-6471-465f-bd08-7f2c164b6755], message
[java.lang.IllegalArgumentException: object mapping [alert] can't be changed
from nested to non-nested]
java.io.IOException: ElasticsearchDao upsert failed: failure in bulk
execution:
[1]: index [metaalert_index], type [null_doc], id
[be44748e-6471-465f-bd08-7f2c164b6755], message
[java.lang.IllegalArgumentException: object mapping [alert] can't be changed
from nested to non-nested]
at
org.apache.metron.elasticsearch.dao.ElasticsearchDao.batchUpdate(ElasticsearchDao.java:368)
at
org.apache.metron.indexing.dao.MultiIndexDao.lambda$batchUpdate$3(MultiIndexDao.java:79)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
",
"fullMessage": "IOException: class
org.apache.metron.elasticsearch.dao.ElasticsearchDao: ElasticsearchDao upsert
failed: failure in bulk execution:
[1]: index [metaalert_index], type [null_doc], id
[be44748e-6471-465f-bd08-7f2c164b6755], message
[java.lang.IllegalArgumentException: object mapping [alert] can't be changed
from nested to non-nested]
java.io.IOException: ElasticsearchDao upsert failed: failure in bulk
execution:
[1]: index [metaalert_index], type [null_doc], id
[be44748e-6471-465f-bd08-7f2c164b6755], message
[java.lang.IllegalArgumentException: object mapping [alert] can't be changed
from nested to non-nested]
at
org.apache.metron.elasticsearch.dao.ElasticsearchDao.batchUpdate(ElasticsearchDao.java:368)
at
org.apache.metron.indexing.dao.MultiIndexDao.lambda$batchUpdate$3(MultiIndexDao.java:79)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"
}
```
Request
```
{
"alertGuids": [
"9b2e9b29-41a5-45fc-aabe-fd49c63af943",
"0fbea2b2-c4f3-43a8-a717-8ceb3467ae10",
"698289ca-43cf-4add-bf35-2a391ae1c244"
],
"metaAlertGuid": "be44748e-6471-465f-bd08-7f2c164b6755",
"sensorTypes": [
"bro"
]
}
```
I'll dig into it a bit, but I assume something isn't quite being
constructed properly.
---
[GitHub] metron pull request #826: METRON-1291: Kafka produce REST endpoint does not ...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/826#discussion_r150653926
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java
---
@@ -108,6 +108,9 @@ public ZkUtils zkUtils() {
producerConfig.put("key.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
producerConfig.put("value.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
producerConfig.put("request.required.acks", 1);
+if
(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY,
Boolean.class, false)) {
+ producerConfig.put("security.protocol", "SASL_PLAINTEXT");
--- End diff --
KAFKA_SECURITY_PROTOCOL is carried through by metron.j2
(`KAFKA_SECURITY_PROTOCOL="{{kafka_security_protocol}}"`). Given that, we
should be fine to use it here, I believe.
---
[GitHub] metron pull request #824: METRON-1289: Alert fields are lost when a MetaAler...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/824#discussion_r150641377
--- Diff:
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/InMemoryMetaAlertDao.java
---
@@ -200,4 +207,23 @@ public MetaAlertCreateResponse
createMetaAlert(MetaAlertCreateRequest request)
createResponse.setCreated(true);
return createResponse;
}
+
+ @Override
+ public boolean addAlertsToMetaAlert(String metaAlertGuid,
Collection alertGuids,
+ Collection sensorTypes) throws IOException {
+return true;
--- End diff --
MetaAlertControllerIntegrationTest would be the best place to add these
integration tests, and I think they should be added to make sure the REST calls
line up the way we do everything else.
---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/824 @merrimanr Can you merge master into this PR? There is at least one fix in here that caused me problems spinning this up that's in master, so I'd like to have it pulled in so testing is as stable as possible. ---
[GitHub] metron pull request #824: METRON-1289: Alert fields are lost when a MetaAler...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/824#discussion_r150239607
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/MetaAlertController.java
---
@@ -60,5 +63,37 @@
) throws RestException {
return new ResponseEntity<>(metaAlertService.create(createRequest),
HttpStatus.OK);
}
+
+ @ApiOperation(value = "Create a meta alert")
+ @ApiResponse(message = "Created meta alert", code = 200)
+ @RequestMapping(value = "/add/alert", method = RequestMethod.POST)
+ ResponseEntity addAlertsToMetaAlert(
+ @ApiParam(name = "request", value = "Meta Alert Create Request",
required = true)
+ @RequestBody final MetaAlertAddRemoveRequest
metaAlertAddRemoveRequest
+ ) throws RestException {
+return new
ResponseEntity<>(metaAlertService.addAlertsToMetaAlert(metaAlertAddRemoveRequest),
HttpStatus.OK);
+ }
+
+ @ApiOperation(value = "Create a meta alert")
+ @ApiResponse(message = "Created meta alert", code = 200)
+ @RequestMapping(value = "/remove/alert", method = RequestMethod.POST)
+ ResponseEntity removeAlertsFromMetaAlert(
+ @ApiParam(name = "request", value = "Meta Alert Create Request",
required = true)
+ @RequestBody final MetaAlertAddRemoveRequest
metaAlertAddRemoveRequest
+ ) throws RestException {
+return new
ResponseEntity<>(metaAlertService.removeAlertsFromMetaAlert(metaAlertAddRemoveRequest),
HttpStatus.OK);
+ }
+
+ @ApiOperation(value = "Create a meta alert")
+ @ApiResponse(message = "Created meta alert", code = 200)
+ @RequestMapping(value = "/update/status/{guid}/{status}", method =
RequestMethod.POST)
+ ResponseEntity updateMetaAlertStatus(
+ final @ApiParam(name = "guid", value = "Kafka topic name", required
= true)
+ @PathVariable String guid,
+ final @ApiParam(name = "status", value = "Kafka topic name",
required = true)
+ @PathVariable String status) throws RestException {
+return new
ResponseEntity<>(metaAlertService.updateMetaAlertStatus(guid,
+MetaAlertStatus.valueOf(status.toUpperCase())), HttpStatus.OK);
+ }
--- End diff --
I think we'll also need to be able to add comments and a name to the
metaalert as a whole, to support work in
https://github.com/apache/metron/pull/803/
@iraghumitra Is there anything else needed in terms of updates on meta
alerts themselves (underlying alerts aren't affected by this stuff)?
---
[GitHub] metron pull request #824: METRON-1289: Alert fields are lost when a MetaAler...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/824#discussion_r150230733
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
---
@@ -256,59 +256,91 @@ public Document getLatest(final String guid, final
String sensorType) throws IOE
return ret.orElse(null);
}
+ @Override
+ public Iterable getAllLatest(final Collection guids,
final Collection sensorTypes) throws IOException {
+List documents = searchByGuids(
+guids
+, sensorTypes
+, hit -> {
+ Long ts = 0L;
+ String doc = hit.getSourceAsString();
+ String sourceType =
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+ try {
+return Optional.of(new Document(doc, hit.getId(), sourceType,
ts));
+ } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve latest: "
+ e.getMessage(), e);
+ }
+}
+
+);
+return documents;
+ }
+
+ Optional searchByGuid(String guid, String sensorType,
+ Function<SearchHit, Optional> callback) {
+Collection sensorTypes = sensorType != null ?
Collections.singleton(sensorType) : null;
+List results = searchByGuids(Collections.singleton(guid),
sensorTypes, callback);
+if (results.size() > 0) {
+ return Optional.of(results.get(0));
+} else {
+ return Optional.empty();
+}
+ }
+
/**
* Return the search hit based on the UUID and sensor type.
* A callback can be specified to transform the hit into a type T.
* If more than one hit happens, the first one will be returned.
*/
- Optional searchByGuid(String guid, String sensorType,
+ List searchByGuids(Collection guids, Collection
sensorTypes,
Function<SearchHit, Optional> callback) {
QueryBuilder query;
-if (sensorType != null) {
- query = QueryBuilders.idsQuery(sensorType + "_doc").ids(guid);
+if (sensorTypes != null) {
+ String[] types = sensorTypes.stream().map(sensorType -> sensorType +
"_doc").toArray(String[]::new);
+ query = QueryBuilders.idsQuery(types).ids(guids);
} else {
- query = QueryBuilders.idsQuery().ids(guid);
+ query = QueryBuilders.idsQuery().ids(guids);
}
SearchRequestBuilder request = client.prepareSearch()
--- End diff --
This is only going to return 10 max results, from the default size on the
query. This was fine before (since we only returned one), but it's going to be
incomplete with the refactor.
---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/824 @merrimanr For the findAllLatest, can we add a default implementation to the interface that would cover HBase (i.e. just do a for loop lookup), even if it's not efficient. Otherwise, it should just be a MultiGet right if we want to do something more efficient / HBase specific, right? ---
[GitHub] metron issue #810: METRON-1270 fix for warnings missing @return tag argument...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/810 @dbist Looks like this did get merged in (commit cf404f77d16daca5692f6197575bcc56c58fc2a9), but the PR is still open. I don't see anything wrong with the commit. Can you just close this PR after @cestella confirms we're in the clear (since I think he merged it)? ---
[GitHub] metron issue #812: METRON-1273: Website documentation link should point to t...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/812 +1 @JonZeolla You should be good to go, if you're still going to be the one to push it through. ---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/824 Do we know why the partial update doesn't work? I'm not necessarily opposed to doing this as a short term fix, but I'd like to know root cause. It seems like overkill to submit a full object update on every alert that goes into a metaalert (e.g. that could be a hundred+ full document reindexes happening every time) ---
[GitHub] metron pull request #818: METRON-1284: Remove extraneous dead query in Elast...
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/818 METRON-1284: Remove extraneous dead query in ElasticsearchDao ## Contributor Comments Delete a pointless query. Given that it's essentially just a noop (we query ES and then do nothing with the result other than retrieve a field), there's not really anything to add tests about and it didn't break existing tests. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron remove_dead_code Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/818.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #818 commit 5001a6d6ea282a06831be1f25e7ed21dc8ddef0e Author: justinjleet <justinjl...@gmail.com> Date: 2017-10-26T19:21:23Z Remove extraneous ES call ---
[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/803#discussion_r147152861
--- Diff:
metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts
---
@@ -337,12 +343,67 @@ export class TreeViewComponent extends
TableViewComponent implements OnChanges {
});
}
+ canCreateMetaAlert(count: number) {
+if (count > 999) {
--- End diff --
@james-sirota You'd know a bit more about the practical usage pattern than
I would. Do you have any input on this limit? I'm guessing we could/should
probably drop it a bit to give ourselves a bit more safety factor (e.g. 200?),
assuming that fits a practical pattern.
---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/811 Master is merged in, so this Travis run should be legit ---
[GitHub] metron issue #816: METRON-1280: 0.4.1 -> 0.4.2 missed a couple of projects
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/816 I agree we leave it as-is. ---
[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/803#discussion_r146979225
--- Diff:
metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts
---
@@ -337,12 +343,67 @@ export class TreeViewComponent extends
TableViewComponent implements OnChanges {
});
}
+ canCreateMetaAlert(count: number) {
+if (count > 999) {
--- End diff --
Do we prevent users from adding enough alerts to go over the limit? E.g.
if I have 995 alerts and I try to add 20 more, will the UI prevent it?
---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/811 Travis failure should be resolved once https://github.com/apache/metron/pull/816 is in, and I'll merge in master and push again once it is. ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/811 @nickwallen I added testing around the status stuff, and a couple docs. Let me know if there's anything else that should be updated (or isn't clear or whatever else). ---
[GitHub] metron pull request #811: METRON-1272: Hide child alerts from searches and g...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/811#discussion_r146965770
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
---
@@ -295,19 +341,199 @@ protected Document
buildCreateDocument(MultiGetResponse multiGetResponse, List index)
throws IOException {
-// We have an update to a meta alert itself
-// If we've updated the alerts field (i.e add/remove), recalculate
meta alert scores.
+ protected void handleMetaUpdate(Document update) throws IOException {
--- End diff --
I added a comment on https://github.com/apache/metron/pull/803/, because
the 999 limit is actually only established there, it's not something in master.
---
