GitHub user justinleet opened a pull request:
https://github.com/apache/metron/pull/734
METRON-1158: Build backend for grouping alerts into meta alerts
## Contributor Comments
The motivation for this is also in the ticket, but the idea is that we want
to be able to group alerts into a meta alert that lives and is queryable
alongside the alerts from the standard sources. These would be manually
created, particularly from the UI, as a result of investigations and slicing
and dicing done there.
A REST API endpoint allows for the creation of a meta alert and another
allows for retrieving all meta alerts associated with a given alert (by GUID).
The first is obviously directly useful, and the second is more useful for
testing and validation, but may be useful in other contexts (e.g. finding
events that tend to get swept up into sets of investigations).
The various search endpoints should work as expected, and be able to query
on the metaalerts index.
A test plan is incoming, and a couple more things need to be checked out a
bit more thoroughly, but it should be pretty set for review. If anyone is
interested, it should be easy enough to spin up, use the REST APIs (both the
couple small meta alert ones along with the other search ones) and
interrogation of ES to see what's going on. It's not completely transparent,
because searches against the child alerts of meta alerts need to be done
against the `alert.<field>`, e.g. ` "query":
"alert.guid:35adafd5-ffd1-4b80-806b-91d336b87220 OR
guid:35adafd5-ffd1-4b80-806b-91d336b87220"`
The basic test plan outline is going to be along the lines:
Spin up full dev
Create a couple meta alerts through the REST API.
Ensure they come back as expected from queries.
Ensure they link up appropriately as seen from the other endpoint.
Ensure the other endpoints still work as expected (e.g. search should still
work, etc.)
Documentation also needs some updating and will be added (but I'll
obviously take into account anything anybody wants as I write more of it).
Notes
* Given that these are manually created, a single index is used, rather
than a timed index like for the standard alerts
* Creation of meta alerts takes a map from GUID -> Index. This is so we
can do a multiget on the items, rather than trying to search through every
index for them.
* A list of groups is provided as metadata for the UI to be able to
reproduce whatever slicing and dicing was used to group these alerts into a
metaalert. This could be extended to take arbitrary metadata, but that seems
like followon. It's possible we may want to do something akin to
https://github.com/apache/metron/pull/702, but I'm not familiar enough with it
to say what the benefits of any synergy are.
* Unfortunately, sorting has to be done on a common field. This means the
metadata index is required, at index time, to populate the common field. In
our case, we use the standard threat score field. By default, the meta alerts
will use the sum of the child alerts threat scores, but this can also be set
via configuration to "median", "min", "max", "average", and "count". All
values are calculated and stored in the event that they have other interest,
but can't comingle with standard alerts.
* It's possible that scripted sorting could be used to get around that
limitation, but that's a potentially much larger change that interferes with
existing sorting and requires appropriate construction of the script to handle.
Plus, scripting generally isn't enabled by default for security reasons, iirc.
* The templates have an "alert" nested field added to them. To my
incredible annoyance, this version of elasticsearch will fail on any searches
where "alert" is missing rather than just consider it not found. Even though
it's in a "should" clause. This is improved on in 5.x, where there's a param
to avoid failure. The fields should be removed again as part of that upgrade.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/metron meta_backend
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/734.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #734
----
commit 09c009d3530ead5d5ef7ac7d1ac4e57f39592658
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-08-03T11:59:01Z
Meta Alerts WIP
commit 274a49f453cdccfce5251eb78aa2c51216b5faf2
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-08-30T01:47:32Z
Trying to fix integration test
commit 0bdf67a3103a4a44b206c6ae4147738c807ba7ba
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-08-30T11:21:29Z
test fixes
commit 8fa96167a271551e29f29a4d2e427e87151bf2a6
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-08-30T11:47:36Z
Handling meta implementation in config
commit 4806f6ab0905b093b0e97b7a7892903a4dac4319
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-09-06T01:27:07Z
Removing extraneous notes file and adding README section
commit 01cc5009304a653e39e32495ab32426c01193018
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-09-06T02:05:10Z
Misc cleanup
commit 4705fe83790dcae80ea0d6270f1d2af2d2e0e4ac
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-09-06T04:52:50Z
Fixes
commit ad4e76d91f18c1181abe3ad741fe52ae252ce68d
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-09-06T05:28:33Z
Updating templates to resolve a problem I foresaw then forgot. Also
passing through another function in meta dao
commit 5aec2fbae9034227bd0ea7c38f6d4c479dd0283a
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-09-06T12:49:49Z
Adjusting sorting
commit 5d2ef582681ef3c5e2d041c2d7c2e6dc47ec47e3
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-09-06T13:12:00Z
Updating mapping and index config
----
---
