GitHub user ottobackwards reopened a pull request:
https://github.com/apache/metron/pull/856
METRON-1339 Stellar Shell functionality to verify stored stellar statements
This will allow users to check their deployed statements, say after
upgrade, when they are at rest ( and would fail on use ).
In other words, they were valid when stored, but are not now because of
stellar changes, such as new keywords.
The interface `StellarConfiguredStatementReporter`, which is
`@IndexSubclasses` ( ClassIndex) marked, allows the shell to discover reporters
that can provide statements for validation. This discovery allows de-coupling
of stellar and 'hosts' that know about the location of the stored statements,
and the configuration structure details.
> We do mention the configurations in the shell output at this time.
`metron-common` implements this interface, and can run through visiting all
the configurations.
A new magic keyword was added ` %validate_configured_expressions`
When executed, the shell
- discovers the reporters through class index
- visits the reports, with callbacks for visits or errors
- per visit ( which is called for a specific stellar statement ) the
statement is compiled and errors reported
- if the entire config fails ( threat triage stellar errors fail on
deserialize so we don't get to do ANY enrichment visits in that case ) the
error callback handles that
I'm getting this out there, still a couple of things todo:
[x] ~~full dev run. I have been testing with stellar external to full dev
iteratively~~
[x] ~~readme~~
[x] ~~steps to test~~
[x] ~~unit test~~
[x] ~~ThreatTriage Rule Reason~~
## Testing
- deploy full dev
- edit the squid parser transformation(s) such that the stellar would not
compile, such as adding a dangling `=` in zookeeper
```json
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "squid",
"parserConfig": {
"grokPath": "/patterns/squid",
"patternLabel": "SQUID_DELIMITED",
"timestampField": "timestamp"
},
"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "full_hostname", "domain_without_subdomains" ]
,"config" : {
"full_hostname" : "URL_TO_HOST(url) ="
,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
}
}
]
}
```
- edit the snort threat triage rules in it's enrichment config in zookeeper
( here with an extra `)` )
```json
{
"enrichment" : {
"fieldMap":
{
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel" : {
"fieldMap":
{
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap":
{
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
},
"triageConfig" : {
"riskLevelRules" : [
{
"rule" : "not(IN_SUBNET(ip_dst_addr, '192.168.0.0/24')) )",
"score" : 10
}
],
"aggregator" : "MAX"
}
}
}
```
## Working with zookeeper
I am not a zk cli maestro, so I took the easy way out and used
[ZK-WEB](https://github.com/qiuxiafei/zk-web).
Following the readme instructions it was very simple to clone, edit the
config for full dev, and run from source. If you log in with the creds in the
config you can edit the nodes.
## Results
When you run the magic command, it will report the failed stellar
statements, and the failed enrichment config:
```bash
[Stellar]>>> %validate_configured_expressions
Discovered 1 reporters
Visiting all configurations. ThreatTriage rules are checked when loading
the configuration, thus an invalid ThreatTriage rule will fail the entire
Enrichement Configuration.
Apache Metron
Visiting Apache Metron
==================================================
validating Apache Metron->PARSER->squid->full_hostname
[!] Error Visiting Apache Metron->PARSER->squid->full_hostname
Syntax error @ 1:17 token recognition error at: '='
--
[!] : URL_TO_HOST(url) =
==================================================
==================================================
validating Apache Metron->PARSER->squid->domain_without_subdomains
==================================================
[!] Configuration Apache Metron->ENRICHMENT->snort is not valid, please
review
Done validation
[Stellar]>>>
```
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/metron
stellar_verify_deployed_shell
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/856.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #856
----
commit a5087f3a170eeda6ee778397c919d9eddd5597e2
Author: Otto Fowler <[email protected]>
Date: 2017-11-30T14:15:40Z
Stellar shell functionality to verify stellar statements.
This will allow users to check their deployed statements, say after
upgrade, when they are at rest ( and would fail on use ).
In other words, they were valid when stored, but are not now because of
stellar changes, such as new keywords.
The interface StellarConfiguredStatementReporter, which is @IndexSubclasses
marked, allows the shell to discover
reporters that can provide statements for validation. This discovery
allows de-coupling of stellar and 'hosts' that
know about the location of the stored statements, and the configuration
structure details.
We do mention the configurations in the shell output at this time.
metron-common implements this interface, and can run through visiting all
the configurations.
commit 96df802318d74bf8dfcd3bcae9208f63c3d034f0
Author: Otto Fowler <[email protected]>
Date: 2017-11-30T16:25:58Z
add readme and remove some newlines
commit dcd55e8f4a72e5c3e694807e13c7eebc53d1860f
Author: Otto Fowler <[email protected]>
Date: 2017-11-30T23:34:38Z
add tests for StellarStatementReporter
commit c0315b8291557de94dcf701d8def16d0b2866798
Author: Otto Fowler <[email protected]>
Date: 2017-12-01T01:18:45Z
refactor to utility classes, first step in major refactor
commit 65278a67a07f1c4c23ab2d95ebb6de92e1cac731
Author: Otto Fowler <[email protected]>
Date: 2017-12-01T16:46:57Z
Refactor based on review and inspiration from review.
Although the original implementation was functional, it required
maintainence to keep current.
The suggested 'best state' was to have it be possible, maybe through
annotations, for the validation
system to be able to handle any config, regarless or composition using
annotations.
That would leave it up to the implementor to propertly annotate thier
configurations, and allow for support of new fields.
This is an implementation of that.
I have refactored the implemenations and details, but kept the discovery
and mechanics ( loading and visitation ) somewhat the same.
Hopefully keeping the good and reworking to a more sustainable solution.
Several annotations where created to marks ceratin stellar configruation
objects or scenarios.
A holder object, to hold the configuration object, but knows how to process
the annotations and run the visitation was added.
This holder object and the annotations have parameters and handling for
several special scenarios, such as 2x nested maps.
This implementation should facilitate follow on work to validate files and
streams and blobs by using implementing the StellarValidator interface
and re-using the holder concept ( replacing the providers )
commit 70de632ee583b45c028b23d8305d76e4b6bc70c5
Author: Otto Fowler <[email protected]>
Date: 2017-12-01T16:59:39Z
fix imports
commit 8726a15a3db35bf24408e723ec069a391df16820
Author: Otto Fowler <[email protected]>
Date: 2017-12-01T19:53:52Z
small refactor and javadoc work
commit a6a9a4e5d558209175a8e5d2fa532c845efa830d
Author: Otto Fowler <[email protected]>
Date: 2017-12-03T13:22:50Z
format and javadoc
commit 3f12c2dace1157ffd4e870df39864b61b71c1270
Author: Otto Fowler <[email protected]>
Date: 2017-12-03T20:31:30Z
refactor name and tests
commit 5516bad34573ef11dc40eb9ed23b241e7d84c75f
Author: Otto Fowler <[email protected]>
Date: 2017-12-04T00:54:08Z
fix for exception change
commit b3e7cfb8ac76d618d35da97895169b9069e7fba0
Author: Otto Fowler <[email protected]>
Date: 2017-12-04T01:17:56Z
fix regression after fixing mapping in prior commit
commit c067c9b8e39b91556790645bdae7b3f55d89d6eb
Author: Otto Fowler <[email protected]>
Date: 2017-12-04T16:18:30Z
Merge remote-tracking branch 'apache/master' into
stellar_verify_deployed_shell
commit a814a0e0e497bbb1e45b4694b78d585136c756c8
Author: Otto Fowler <[email protected]>
Date: 2017-12-05T16:12:48Z
Merge remote-tracking branch 'apache/master' into
stellar_verify_deployed_shell
commit 7b28be6e2da9756a3cb4f3234308118e99a2e17c
Author: Otto Fowler <[email protected]>
Date: 2017-12-07T11:55:05Z
Merge remote-tracking branch 'apache/master' into
stellar_verify_deployed_shell
----
---
