GitHub user justinleet reopened a pull request:
https://github.com/apache/metron/pull/811
METRON-1272: Hide child alerts from searches and grouping if they belong to
meta alerts
## Contributor Comments
Adds the ability to hide alerts already contained in a meta alert from
searches. It'll also hide it from the group mechanism (as they've already been
sliced and diced).
This happens by adding a "metaalerts" field to our alerts, which is a
simple list of GUIDs. This does require partial updates to alerts when a meta
alert is created or has the status changed from active to inactive or vice
versa. Alerts can potentially be updated when the "alert" field of a meta
alert is changed (i.e. an alert is added or removed), but this should be
limited to alerts that are actually affected. Given that there are practical
limits (iirc, ~1000) alerts that can be in a given meta alert and this is a
manual action, it shouldn't be particularly bad.
A batchUpdate is added to the IndexDao to avoid making a ton of updates to
ES. It's added accordingly through the implementors (HBase just uses the
List<Put> method, if we want something else let me know, I haven't touched that
stuff in awhile). ElasticsearchMetaAlertDao throws an
UnsupportedOperationException both to avoid having to implement it and also
because creating meta alerts should generally be a single action as a result of
manual intervention, not a bulk operation.
Right now, there is an update to the templates to make them "not_analyzed".
This is unfortunately due to our GUIDs being analyzed fields. I could
potentially delete and reinsert the hyphens appropriately if we care to avoid
that and are willing to absorb a bit of a kludge.
I owe a test plan for this, along with a couple unit tests (particularly
~around grouping, and~ shifting the status of the meta alert and ensuring
queries work as expected). Having said that, it should be fairly testable now
on full dev, just by creating a couple meta alerts with a couple child alerts
and running appropriate searches.
Also definitely owe some documentation, particularly if we need to keep the
field in the templates (and add it as a requirement).
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/metron meta_hiding
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/811.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #811
----
commit b03267b6a0aa71fabedcc0fed1f768b64676b914
Author: justinjleet <[email protected]>
Date: 2017-10-17T11:26:57Z
Untested hiding in both grouping and search
commit e54865e203a57111e2f2485a2e6e0d49e0e9fd2a
Author: justinjleet <[email protected]>
Date: 2017-10-17T13:37:18Z
better
commit 58f1edba1d900d7e66687ee1a05e08cae95b4663
Author: justinjleet <[email protected]>
Date: 2017-10-17T15:16:40Z
Maybe fixing creating meta alert
commit 7e97694fa8eb3a750d2daaf2d41bc52a2150fdb0
Author: justinjleet <[email protected]>
Date: 2017-10-17T16:10:42Z
Adding an alert should be working now
commit 28bb5c13199a483d9c8c51bb782293050a429fe9
Author: merrimanr <[email protected]>
Date: 2017-10-17T16:15:24Z
initial commit
commit 3aa4c158c11c1a72e2e80a9fa2e98a6f1be57348
Author: merrimanr <[email protected]>
Date: 2017-10-17T16:16:07Z
Merge remote-tracking branch 'mirror/master' into METRON-1255
commit bc923540e3a33225d458db9505045cf062d553f2
Author: merrimanr <[email protected]>
Date: 2017-10-17T18:02:49Z
resolved merge conflicts
commit 76f29057ecbd694f513ab6083e4ef921882f9944
Author: justinjleet <[email protected]>
Date: 2017-10-17T22:37:51Z
create alert updates happen in batch. Not adding calculations on, though
commit 7508e111f27a890c686b814450c57a17265ae77d
Author: justinjleet <[email protected]>
Date: 2017-10-17T23:57:11Z
Fixing create
commit 646b8d6bfae681dabfd5f9b192633593f54fe5cd
Author: justinjleet <[email protected]>
Date: 2017-10-18T02:26:59Z
Moving bulkUpdate to interface. Fixing other bugs
commit bfe179bbe09f24e7be7edf4d87b45aef88cf7805
Author: justinjleet <[email protected]>
Date: 2017-10-18T02:57:25Z
refactoring
commit b186a116b5c00af0e104728b0040b0e330eb2881
Author: justinjleet <[email protected]>
Date: 2017-10-18T03:43:22Z
refactoring
commit 05cc0cc1368fbbc5708ae09404acfddf1606ee04
Author: justinjleet <[email protected]>
Date: 2017-10-18T03:53:00Z
Merge branch 'master' into meta_hiding
commit cb6569b9346a319f55d3df12b91c310d402c257c
Author: justinjleet <[email protected]>
Date: 2017-10-18T12:02:29Z
Merge branch 'METRON-1255' into meta_hiding
commit 22896fb841bd6831876352d75572037f147117ba
Author: justinjleet <[email protected]>
Date: 2017-10-18T13:37:26Z
Renaming method from bulk to batch
commit 22e7462f2a6cae917bffc5c4b39f10b579848e63
Author: merrimanr <[email protected]>
Date: 2017-10-18T19:53:43Z
initial commit
commit 319e130d13ca58bb6ebead96250fed9a7fe5cfae
Author: merrimanr <[email protected]>
Date: 2017-10-18T21:28:04Z
Merge remote-tracking branch 'mirror/master' into metaalert-alert-update
# Conflicts:
#
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
#
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
commit 7bb7d946fc1b400a3ff0ed1a79a029650609387a
Author: merrimanr <[email protected]>
Date: 2017-10-18T21:43:28Z
fixed a couple findUpdatedDoc statements
commit b926bfd99560dbbfc3fbd6c88a00e2b3ccee6515
Author: justinjleet <[email protected]>
Date: 2017-10-19T12:56:33Z
remove comment
commit b5290d1b6e0e295c35ffd4e283c23245896a9fbe
Author: justinjleet <[email protected]>
Date: 2017-10-19T14:06:27Z
Changing group to just hide alerts in a meta alert
commit 395a5a6e546e8539fc191a07f93aa48517d4acc8
Author: justinjleet <[email protected]>
Date: 2017-10-19T14:11:50Z
Merge branch 'master' into meta_hiding
commit e669a7f98a42dfec8d70621f299f347d3566a110
Author: justinjleet <[email protected]>
Date: 2017-10-20T19:28:57Z
Merge branch 'METRON-1262' into meta_hiding
commit 860cb846140a91ad028623f14e6c2d545f89905e
Author: justinjleet <[email protected]>
Date: 2017-10-21T22:28:01Z
Finally get unit tests working. At least it was a unit test problem
commit 68bce64dfd6c1674464d7c9a57e2685d034a0644
Author: justinjleet <[email protected]>
Date: 2017-10-23T11:29:18Z
Fixing / adjusting test and apparently lost the hiding in a merge
commit ff99ad1730a5b650e02d6c02433915666664a35a
Author: justinjleet <[email protected]>
Date: 2017-10-23T11:34:19Z
Improving search test
commit 00203d26f928afd613744af6885c03be6a057de3
Author: justinjleet <[email protected]>
Date: 2017-10-23T11:35:44Z
Adding comment
commit 3061ecdc6c19c223e5dddfef9f7b412fdbcd7fce
Author: justinjleet <[email protected]>
Date: 2017-10-23T11:53:02Z
Merge branch 'master' into meta_hiding
commit 1680c75980ee8a94a76cf90cfec570080939e519
Author: justinjleet <[email protected]>
Date: 2017-10-23T12:20:41Z
Don't break things when merging
commit 664cc830c7ef137f0abf5dcb910279813b49bfd1
Author: justinjleet <[email protected]>
Date: 2017-10-23T12:54:16Z
Grouping test
commit 4f60d52093d39a42df2e6ecccc5c8a77182595a7
Author: justinjleet <[email protected]>
Date: 2017-10-23T18:34:27Z
Changing ES component to actually clean up after itself during reset
----
---
