Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/840
Ok, here is the remaining bit of the test plan for verifying ES. Thanks
@cestella for pulling together some of the public test scripts around this.
# Test Script
Testing Kibana dashboards - see
[here](https://github.com/apache/metron/pull/840#issuecomment-348085037)
Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).
# Preliminaries
Set an environment variable to indicate `METRON_HOME`:
* `export METRON_HOME=/usr/metron/0.4.2`
# Deploy the dummy parser
* Edit `$METRON_HOME/config/zookeeper/parsers/dummy.json`:
```
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"dummy"
}
```
* Create the dummy kafka topic:
`/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181
--create --topic dummy --partitions 1 --replication-factor 1`
* Persist config changes: `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i
$METRON_HOME/config/zookeeper -z node1:2181`
* Start via `$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z
node1:2181 -s dummy`
# Send dummy data through
* Edit `~/msg.json` with the following content:
```
{ "guid" : "guid0", "sensor.type" : "dummy", "timestamp" : 100 }
```
* Send `msg.json` through to kafka via `cat ~/msg.json |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
node1:6667 --topic dummy`
* Validate data has been written to the index:
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty'
```
## Test Case: Update via patch
* Patch the message in ES and create a new field 'project' by executing
the following:
```
curl -u user:password -X PATCH --header 'Content-Type: application/json'
--header 'Accept: */*' -d '{
"guid" : "guid0",
"sensorType" : "dummy",
"patch" : [
{
"op": "add"
, "path": "/project"
, "value": "metron"
}
]
}' 'http://node1:8082/api/v1/update/patch'
```
* Validate that the message has a field 'project':
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
"_source" : [ "project" ]
}
'
```
## Test Case: Update via replace
* Replace the message in ES and create a couple of modifications:
* new field `new_field` == "brand new"
* modified `timestamp` == 7
Execute the following:
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: */*' -d '{
"guid" : "guid0",
"sensorType" : "dummy",
"replacement" : {
"source:type": "dummy",
"guid" : "guid0",
"new_field" : "brand new",
"timestamp" : 7
}
}' 'http://node1:8082/api/v1/update/replace'
```
* Validate that the message has a field 'new_field':
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
"_source" : [ "new_field", "timestamp" ]
}
'
```
## Meta Alerts Test
### Set Up Base Data
We're going to set up a bit of base data.
Retrieve the current list of indices so we know where to put our data
```
curl 'node1:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted
store.size pri.store.size
green open snort_index_2017.09.06.14 1 0 130 0
180.9kb 180.9kb
green open bro_index_2017.09.06.14 1 0 160 0
564.3kb 564.3kb
green open .kibana 1 0 52 0
71.2kb 71.2kb
green open metaalert_index 1 0 6 0
62.3kb 62.3kb
```
In this case, we care about `snort_index_2017.09.06.14` and
`metaalert_index`. To make our lives easier, we'll add a couple of stripped
down messages to the our snort index (Make sure to sub in the correct index
name):
```
curl -XPUT
'node1:9200/snort_index_2017.09.06.14/snort_doc/snort_test_1?pretty' -H
'Content-Type: application/json' -d'
{
"msg": "snort test alert",
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000,
"threat:triage:rules:0:reason": null,
"threat:triage:score": 10,
"is_alert": "true",
"ip_src_port": "50187",
"guid": "snort_test_1"
}
'
```
```
curl -XPUT
'node1:9200/snort_index_2017.09.06.14/snort_doc/snort_test_2?pretty' -H
'Content-Type: application/json' -d'
{
"msg": "snort test alert 2",
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000,
"threat:triage:rules:0:reason": null,
"threat:triage:score": 10,
"is_alert": "true",
"ip_src_port": "50187",
"guid": "snort_test_2"
}
'
```
### Create a Meta Alert
At this point, we'll group these alerts together. In Ambari, go to Metron
-> Quick Links -> Swagger UI and go to the "Meta Alert Controller".
Click on the create endpoint, and we'll want to send this request (but with
the appropriate index subbed in):
```
{
"groups": [
"group_one",
"group_two"
],
"guidToIndices": {
"snort_test_1":"snort_index_2017.09.06.14",
"snort_test_2":"snort_index_2017.09.06.14"
}
}
```
Validate that the metaalert was created and looks good:
```
curl 'node1:9200/m*/_search?pretty'
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "metaalerts",
"_type" : "metaalert_doc",
"_id" : "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
"_score" : 1.0,
"_timestamp" : 1504725383439,
"_source" : {
"average" : 10.0,
"min" : 10.0,
"median" : 10.0,
"alert" : [ {
"msg" : "snort test alert",
"threat:triage:rules:0:reason" : null,
"ip_dst_port" : "8080",
"ethsrc" : "0A:00:27:00:00:00",
"threat:triage:score" : 10,
"is_alert" : "true",
"protocol" : "TCP",
"source:type" : "snort",
"ip_dst_addr" : "192.168.66.121",
"ip_src_port" : "50187",
"guid" : "snort_test_1",
"ip_src_addr" : "192.168.66.1",
"threat:triage:rules:0:score" : 10,
"timestamp" : 1504708744000
}, {
"msg" : "snort test alert 2",
"threat:triage:rules:0:reason" : null,
"ip_dst_port" : "8080",
"ethsrc" : "0A:00:27:00:00:00",
"threat:triage:score" : 10,
"is_alert" : "true",
"protocol" : "TCP",
"source:type" : "snort",
"ip_dst_addr" : "192.168.66.121",
"ip_src_port" : "50187",
"guid" : "snort_test_2",
"ip_src_addr" : "192.168.66.1",
"threat:triage:rules:0:score" : 10,
"timestamp" : 1504708744000
} ],
"max" : 10.0,
"threat:triage:score" : 20.0,
"count" : 2,
"guid" : "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
"groups" : [ "group_one", "group_two" ],
"sum" : 20.0,
"status" : "active"
}
} ]
}
}
```
Note that both sub alerts are present, the various counts are filled in, a
GUID has been given specifically to this meta alert, etc.
### Searching
Searching from the REST API works mostly as expected. After the above data
has been created, use the search endpoint to run this query:
```
{
"from": 0,
"size": 2,
"indices": [
"*"
],
"query": "guid=snort_test_2",
"sort": [
{
"field": "_timestamp",
"sortOrder": "ASC"
}
]
}
```
The result should look similar to the following, and have the messages we'd
expect (one from snort and one from metaalerts):
```
{
"total": 2,
"results": [
{
"id": "snort_test_2",
"source": {
"msg": "snort test alert 2",
"threat:triage:rules:0:reason": null,
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"threat:triage:score": 10,
"is_alert": "true",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_port": "50187",
"guid": "snort_test_2",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000
},
"score": 1,
"index": "snort_index_2017.09.06.14"
},
{
"id": "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
"source": {
"average": 10,
"min": 10,
"median": 10,
"alert": [
{
"msg": "snort test alert",
"threat:triage:rules:0:reason": null,
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"threat:triage:score": 10,
"is_alert": "true",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_port": "50187",
"guid": "snort_test_1",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000
},
{
"msg": "snort test alert 2",
"threat:triage:rules:0:reason": null,
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"threat:triage:score": 10,
"is_alert": "true",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_port": "50187",
"guid": "snort_test_2",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000
}
],
"max": 10,
"threat:triage:score": 20,
"count": 2,
"guid": "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
"groups": [
"group_one",
"group_two"
],
"sum": 20,
"status": "active"
},
"score": 1,
"index": "metaalerts"
}
]
}
```
---
