GitHub user justinleet opened a pull request:
https://github.com/apache/metron/pull/811
METRON-1272: Hide child alerts from searches and grouping if they belong to
meta alerts
## Contributor Comments
Adds the ability to hide alerts already contained in a meta alert from
searches. It'll also hide it from the group mechanism (as they've already been
sliced and diced).
This happens by adding a "metaalerts" field to our alerts, which is a
simple list of GUIDs. This does require partial updates to alerts when a meta
alert is created or has the status changed from active to inactive or vice
versa. Alerts can potentially be updated when the "alert" field of a meta
alert is changed (i.e. an alert is added or removed), but this should be
limited to alerts that are actually affected. Given that there are practical
limits (iirc, ~1000) alerts that can be in a given meta alert and this is a
manual action, it shouldn't be particularly bad.
A batchUpdate is added to the IndexDao to avoid making a ton of updates to
ES. It's added accordingly through the implementors (HBase just uses the
List<Put> method, if we want something else let me know, I haven't touched that
stuff in awhile). ElasticsearchMetaAlertDao throws an
UnsupportedOperationException both to avoid having to implement it and also
because creating meta alerts should generally be a single action as a result of
manual intervention, not a bulk operation.
Right now, there is an update to the templates to make them "not_analyzed".
This is unfortunately due to our GUIDs being analyzed fields. I could
potentially delete and reinsert the hyphens appropriately if we care to avoid
that and are willing to absorb a bit of a kludge.
I owe a test plan for this, along with a couple unit tests (particularly
around grouping, and shifting the status of the meta alert and ensuring queries
work as expected). Having said that, it should be fairly testable now on full
dev, just by creating a couple meta alerts with a couple child alerts and
running appropriate searches.
Also definitely owe some documentation, particularly if we need to keep the
field in the templates (and add it as a requirement).
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/metron meta_hiding
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/811.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #811
----
commit b03267b6a0aa71fabedcc0fed1f768b64676b914
Author: justinjleet <[email protected]>
Date: 2017-10-17T11:26:57Z
Untested hiding in both grouping and search
commit e54865e203a57111e2f2485a2e6e0d49e0e9fd2a
Author: justinjleet <[email protected]>
Date: 2017-10-17T13:37:18Z
better
commit 58f1edba1d900d7e66687ee1a05e08cae95b4663
Author: justinjleet <[email protected]>
Date: 2017-10-17T15:16:40Z
Maybe fixing creating meta alert
commit 7e97694fa8eb3a750d2daaf2d41bc52a2150fdb0
Author: justinjleet <[email protected]>
Date: 2017-10-17T16:10:42Z
Adding an alert should be working now
commit 28bb5c13199a483d9c8c51bb782293050a429fe9
Author: merrimanr <[email protected]>
Date: 2017-10-17T16:15:24Z
initial commit
commit 3aa4c158c11c1a72e2e80a9fa2e98a6f1be57348
Author: merrimanr <[email protected]>
Date: 2017-10-17T16:16:07Z
Merge remote-tracking branch 'mirror/master' into METRON-1255
commit bc923540e3a33225d458db9505045cf062d553f2
Author: merrimanr <[email protected]>
Date: 2017-10-17T18:02:49Z
resolved merge conflicts
commit 76f29057ecbd694f513ab6083e4ef921882f9944
Author: justinjleet <[email protected]>
Date: 2017-10-17T22:37:51Z
create alert updates happen in batch. Not adding calculations on, though
commit 7508e111f27a890c686b814450c57a17265ae77d
Author: justinjleet <[email protected]>
Date: 2017-10-17T23:57:11Z
Fixing create
commit 646b8d6bfae681dabfd5f9b192633593f54fe5cd
Author: justinjleet <[email protected]>
Date: 2017-10-18T02:26:59Z
Moving bulkUpdate to interface. Fixing other bugs
commit bfe179bbe09f24e7be7edf4d87b45aef88cf7805
Author: justinjleet <[email protected]>
Date: 2017-10-18T02:57:25Z
refactoring
commit b186a116b5c00af0e104728b0040b0e330eb2881
Author: justinjleet <[email protected]>
Date: 2017-10-18T03:43:22Z
refactoring
commit 05cc0cc1368fbbc5708ae09404acfddf1606ee04
Author: justinjleet <[email protected]>
Date: 2017-10-18T03:53:00Z
Merge branch 'master' into meta_hiding
commit cb6569b9346a319f55d3df12b91c310d402c257c
Author: justinjleet <[email protected]>
Date: 2017-10-18T12:02:29Z
Merge branch 'METRON-1255' into meta_hiding
commit 22896fb841bd6831876352d75572037f147117ba
Author: justinjleet <[email protected]>
Date: 2017-10-18T13:37:26Z
Renaming method from bulk to batch
commit 22e7462f2a6cae917bffc5c4b39f10b579848e63
Author: merrimanr <[email protected]>
Date: 2017-10-18T19:53:43Z
initial commit
commit 319e130d13ca58bb6ebead96250fed9a7fe5cfae
Author: merrimanr <[email protected]>
Date: 2017-10-18T21:28:04Z
Merge remote-tracking branch 'mirror/master' into metaalert-alert-update
# Conflicts:
#
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
#
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
commit 7bb7d946fc1b400a3ff0ed1a79a029650609387a
Author: merrimanr <[email protected]>
Date: 2017-10-18T21:43:28Z
fixed a couple findUpdatedDoc statements
commit b926bfd99560dbbfc3fbd6c88a00e2b3ccee6515
Author: justinjleet <[email protected]>
Date: 2017-10-19T12:56:33Z
remove comment
commit b5290d1b6e0e295c35ffd4e283c23245896a9fbe
Author: justinjleet <[email protected]>
Date: 2017-10-19T14:06:27Z
Changing group to just hide alerts in a meta alert
commit 395a5a6e546e8539fc191a07f93aa48517d4acc8
Author: justinjleet <[email protected]>
Date: 2017-10-19T14:11:50Z
Merge branch 'master' into meta_hiding
commit e669a7f98a42dfec8d70621f299f347d3566a110
Author: justinjleet <[email protected]>
Date: 2017-10-20T19:28:57Z
Merge branch 'METRON-1262' into meta_hiding
commit 860cb846140a91ad028623f14e6c2d545f89905e
Author: justinjleet <[email protected]>
Date: 2017-10-21T22:28:01Z
Finally get unit tests working. At least it was a unit test problem
commit 68bce64dfd6c1674464d7c9a57e2685d034a0644
Author: justinjleet <[email protected]>
Date: 2017-10-23T11:29:18Z
Fixing / adjusting test and apparently lost the hiding in a merge
commit ff99ad1730a5b650e02d6c02433915666664a35a
Author: justinjleet <[email protected]>
Date: 2017-10-23T11:34:19Z
Improving search test
commit 00203d26f928afd613744af6885c03be6a057de3
Author: justinjleet <[email protected]>
Date: 2017-10-23T11:35:44Z
Adding comment
commit 3061ecdc6c19c223e5dddfef9f7b412fdbcd7fce
Author: justinjleet <[email protected]>
Date: 2017-10-23T11:53:02Z
Merge branch 'master' into meta_hiding
----
---
