GitHub user iraghumitra reopened a pull request:
https://github.com/apache/metron/pull/620
Metron-988: UI for viewing alerts generated by Metron
## Contributor Comments
## Description
Metron being a cybersecurity framework has the capability to raise alerts
based on the events it processes. It would be nice to have a GUI to view these
alerts and an option to integrate with any external workflow systems.
A GUI design for viewing the alerts has been discussed in the community a
while back (community discussion link is attached to the JIRA). This PR is a
first pass to have a UI around alerts.
The current implementation is a standalone GUI implementation i.e the GUI
connects to Elastic search for showing the alerts but the GUI stores all other
data in the browser cache. Below are the GUI features and how they manage their
data.
### Features
**List Alerts:** The alert list page displays the alerts generated by the
metron framework in a table.
- By default, the table shows the recent alerts on the top i.e alerts are
sorted descending on timestamp
- The GUI polls for the alerts and the polling interval is configurable
- The polling is paused when user opens any other configuration panes
- The table shows only the columns that could be displayed in its
horizontal view port any additional columns would be hidden. These hidden
columns would be visible if the UI is opened in a display that has a wider view
port
- The table column width is fixed and any text exceeding the allotted width
would be displayed using center ellipses
**Search Alerts:** Users can search for alerts using the search bar above
the alerts table.
- The search query follows lucene syntax
- User can click on any text in the table to add it as a search parameter
- When user mouseovers on a search text a cross icon appears at the end of
search text, clicking on the cross icon would remove the search filter and the
operator following or preceding it
- There is a cross icon on the search box itself that would clear the search
**Configure Table Columns:** When a user selects the gear icon on the table
header a pane appears that lists all the columns available across all the valid
search indexes.
- User can choose the columns that he wishes to view on the UI
- User can configure the order in which the selected columns should appear
in the table (Using the arrow icons)
- User can rename the columns name Ex:
'enrichments:geo:ip_dst_addr:country' can be renamed to 'Dst Country'. This is
just for display convenience and the changes are not propagated to any system
in Metron
**Configure Table Row Settings:** When user selects the 'slides' icon on
top of the table he gets an option to choose the following table settings
- Refresh Interval
- Table Size
- HIDE Resolved Alerts & HIDE Dismissed Alerts, these are non-functional
features and can be used as global filters for filtering alerts that would be
displayed when integrated with any workflow systems
**Save Search:** When a user selects the save button next to the search box
he gets an option to save the current search. The last searched search
parameters can be saved along with the current column configuration. User's
should give a name for the search he wishes to save.
**Saved Searches:** When a user selects the 'Searches' button present
before the search box he can see all his saved searches. Last five executed
searches are also shown in this pane as a convenience. The pane also provides
an option to deleted the saved searches.
| UI Component | API Type
|
| ---------------------------------------- |
---------------------------------------- |
| List Alerts | Fetches data from ES
|
| Configure Table Columns | Fetches all columns names
across all indexes from ES and saves the user-selected columns to browser cache
|
| Search Alerts | Fetches data from ES using
query_string |
| Save Last 5 Recent Searches | Saves data into browser cache
and fetches data from browser cache |
| Saved Searches (Saves search query and column selection) | Saves data
into browser cache and fetches data from browser cache |
| Table Settings - Refresh Interval & Page Size | Saves data into browser
cache and fetches data from browser cache |
*This is a big pull request and I understand that I am asking a lot
already. If reviewers feel they have found a critical bug that should be fixed
before this can be accepted, please let me know I will address it ASAP.
Otherwise, I can create Jiras for bugs/issues that can be done as follow-on
work.*
### Not-Yet-Functional
Alert Status Fields in Table: This field is just an indicator of how we can
integrate with any other workflow management systems
HIDE Resolved Alerts & HIDE Dismissed Alerts in Table Settings: These
fields just show how we can use global filters to filter the data that is
displayed
### Testing the PR
**Prerequisites:**
* Elastic search should be up and running and should have alerts populated
by metron topologies
* The alerts can be populated using Quick Dev, Full Dev or any other setup
* node.JS >= 7.8.0
All the UI features can be verified using any one of the two approaches:
1. Using Development Server
2. Installing on an existing Cluster
**Using Development Server:**
1. Install all the dependent node_modules using the following command
```
cd incubator-metron/metron-interface/metron-alerts
npm install
```
2. UI can be run by using the following command
```
./scripts/start-dev.sh
```
**NOTE**: *In the development mode ui by default connects to ES at
http://node1:9200 for fetching data. If you wish to change it you can change
the ES url at incubator-metron/metron-interface/metron-alerts/proxy.conf.json*
**Installing on an existing Cluster:**
1. Build Metron
```
mvn clean package -DskipTests
```
2. Copy
`incubator-metron/metron-interface/metron-alerts/target/metron-alerts-METRON_VERSION-archive.tar.gz`
to the desired host.
3. Untar the archive in the target directory. The directory structure will
look like:
```
bin
start_alerts_ui.sh
web
alerts-ui
package.json
server.js
web assets (html, css, js, ...)
```
4. [Expressjs](https://github.com/expressjs/express) web server script is
included in the build that will serve the application. Start the application
with the script:
```
./bin/start_alerts_ui.sh
Usage: server.js -p [port] -r [restUrl]
Options:
-p Port to run metron alerts ui [required]
-r, --resturl Url where elastic search rest api is available
[required]
```
### UI Automation Testing
The UI has e2e tests written using protractor. These test cases can be
extended as per need. The steps to run the e2e test are as follows.
An expressjs server is available for mocking the elastic search api.
1. Run e2e webserver :
```
cd incubator-metron/metron-interface/metron-alerts
sh ./scripts/start-server-for-e2e.sh
```
2. run e2e test using the following command
```
cd incubator-metron/metron-interface/metron-alerts
npm run e2e
```
### Next
As stated this is a first pass for building a UI for alerts. The following
could be the next possible steps.
- A Rest based middleware to abstract the GUI from the alerts repository.
metron-rest is a good candidate to host such rest api's
- More features like faceting, grouping can be added to the GUI
- Improvements to the existing search mechanism to make it simpler
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/iraghumitra/incubator-metron METRON-988
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/620.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #620
----
commit e22a82ec19d3e48b1dca629eaea4afcefa22acf1
Author: iraghumitra <[email protected]>
Date: 2017-05-23T12:40:41Z
Initial commit for alerts
commit 38c3abb7222251a5c97c4936ab16eec9ddf5eca2
Author: iraghumitra <[email protected]>
Date: 2017-05-23T12:43:26Z
Formatted README
commit b5e1e10091c6d77d447334069d17997a33d50a80
Author: iraghumitra <[email protected]>
Date: 2017-05-23T12:50:54Z
Added TOC and all sections i wanted
commit ad34ab1377889c8b47a70b9e65d1e760f20ffd1e
Author: iraghumitra <[email protected]>
Date: 2017-05-24T12:50:44Z
Added script to deploy on cluster updated readme for the same
Integrated pom.xml to build metron alerts
commit ca7d317735b152ec7d6fa50b2791261a63c84829
Author: iraghumitra <[email protected]>
Date: 2017-06-01T11:26:20Z
- Added displayQuery to show renamed cols in search bar
- Added translate pipe to rename col names
- Added option to rename col's
commit e96d7b9b0947b50c56e0a557051da094abe0c3b9
Author: iraghumitra <[email protected]>
Date: 2017-06-01T11:27:17Z
Added missed file
commit c0b5ffbbf2d000892cc74e50492a7ec5be1e2fec
Author: RaghuMitra <[email protected]>
Date: 2017-06-01T11:28:33Z
Merge pull request #1 from iraghumitra/rename-cols
Rename cols
commit 5a5cb1876c72924c39426ac9e019b445db538e23
Author: iraghumitra <[email protected]>
Date: 2017-06-01T12:03:50Z
Added missing pipe
commit 9a15b3d7d1327032ee6be91200434a10e9cbdfc9
Author: iraghumitra <[email protected]>
Date: 2017-06-01T14:06:56Z
- Indent space in search bar add 10 to 15 pixels of indent on the left and
right side of the textbox
- Take out trashcan for recent
- Change recent to 10 by default but store 25, Iâll do a design for how
that needs to look
- when the search box overflows, the save search button should be centered
vertically with the expanded textbook
- Search button should be square
commit 8962211a3a557d5d371e63da19a580c3ef0926e8
Author: iraghumitra <[email protected]>
Date: 2017-06-08T07:24:52Z
1. Removed the filters from list page
2. Corrected the mapping for alert id
3. Fixed issue with severity colour coding, the color coding was
disappearing if we hover in and hover out of the severity cell
4. Moved alert-severity.directive to directives folder
5. Removed bootstrap from angular-cli as this is already included by ng
build
commit 17b1e3a507fcf3260e461bb9dd4451301e5420e1
Author: iraghumitra <[email protected]>
Date: 2017-06-19T07:39:01Z
Merge branch 'master' into METRON-988
commit 9dd171a73aaa4af126d68e351e2b5cda03fb7676
Author: iraghumitra <[email protected]>
Date: 2017-06-19T13:30:07Z
Added e2e test cases
Added license header to all files
commit 3d34df625fd9cd6d0f449732a1ba65bbec9fbe94
Author: iraghumitra <[email protected]>
Date: 2017-06-19T14:50:00Z
Added licence header to all files
Removed failing text
commit 8375ce94feb71993aa90ad788cfcb99719c91391
Author: iraghumitra <[email protected]>
Date: 2017-06-19T14:58:34Z
Updated Readme for e2e
commit 979469c851b1b060379b2117cdaceacf5ada8e92
Author: iraghumitra <[email protected]>
Date: 2017-06-19T15:52:09Z
Update version in package.json
Renamed mock folder to mock-data
commit 4d7109d2e7d3038a11e06356ed411e214247ef26
Author: iraghumitra <[email protected]>
Date: 2017-06-19T16:06:05Z
Made lint happy
commit 5d24357de83b110195a9e78566fabfac23e342fd
Author: iraghumitra <[email protected]>
Date: 2017-06-19T17:12:34Z
Fixed brokenlink in readme
commit 28ce6f60af63763be4093315c397f31a236a6cf5
Author: iraghumitra <[email protected]>
Date: 2017-06-19T17:14:10Z
Fixed brokenlink in readme
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
