GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/832
METRON-1301 Sorting on Triage Score Unexpectedly Filters Some Recods
This PR makes the following changes.
* Adds logic to sort fields that are not present in all of the underlying
indices. For example, threat triage score is only present in the records that
have been triaged. Without this PR, sorting on the threat triage score would
unexpectedly filter the search results.
* When sorted descending, missing fields are sorted last.
* When sorted ascending, missing fields are sorted first.
* Adds validation and error checking of the response received from
Elasticsearch. Previously there was little to no validation of the response.
This also handles partial shard failures; errors that occur only on a subset of
the data. Without this change some errors that occurred within Elasticsearch
were not visible.
* Adds additional logging in the ElasticsearchDAO so that errors can be
traced and debugged.
* Refactored the ElasticsearchDAO and extracted some logic into two new
interfaces. This was done so that I could write tests for the new logic. It
was not possible to specifically test this change without refactoring due to
limitations with mocking.
* The column metadata logic is now contained in `ColumnMetadataDao` and
`ElasticsearchColumnMetadataDao`. Without pulling this logic out, it was not
possible to test the new sorting logic.
* The logic for submitting a request to Elasticsearch and validating
the response is now contained within `ElasticsearchSearchSubmitter` class.
Without refactoring this it was not possible to test the new response handling
logic.
## Testing
1. Launch Full Dev
2. Allow some alerts to accumulate so that when you open the Alerts UI
there are a fair number of alerts to work with.
3. By default it will sort by timestamp. Take note of the number of
records returned.
4. Click on the "Score" field to sort the results by threat triage score.
5. Note the total number of records returned after sorting. This should be
equal to or greater than what you saw before.
6. The highest scores should be sorted first. Records that do not contain
a threat triage score should be sorted last.
7. Click on the "Score" field to reverse the sort order. Records without a
threat triage score should now be sorted first.
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1301
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/832.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #832
----
commit de7eba97331442fac293c5c3ca72993f7a431200
Author: Nick Allen <n...@nickallen.org>
Date: 2017-10-30T16:02:46Z
METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters Some
Records
----
---
