GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/138
METRON-189: Add the ability to do global validations on messages passing
through the parser.
Allow the user to specify field level or message level validations to
ensure messages coming from the parser are valid. For instance, allow the
ability ensure that a field is an IPv4 address.
If a field is invalid, send to a separate stream from the parser bolt.
Follow-on work should be done to send this stream to the index for
after-the-fact inspection.
I added the following validation functions:
* `MQL` : Execute a Query Language statement. Expects the query string in
the `condition` field of the config.
* `IP` : Validates that the input fields are an IP addres. By default, if
no configuration is set, it assumes `IPV4`, but you can specify the type
by passing in the config by passing in `type` with either `IPV6` or `IPV4`.
* `DOMAIN` : Validates that the fields are all domains.
* `EMAIL` : Validates that the fields are all email addresses
* `URL` : Validates that the fields are all URLs
* `DATE` : Validates that the fields are a date. Expects `format` in the
config.
* `INTEGER` : Validates that the fields are an integer. String
representation of an integer is allowed.
* `REGEX_MATCH` : Validates that the fields match a regex. Expects
`pattern` in the config.
* `NOT_EMPTY` : Validates that the fields exist and are not empty (after
trimming.)
Because of the nice overlap, I also added these functions to the query
language, so query language rules can take advantage of `IS_IP(field1)` for
instance.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron validation
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/138.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #138
----
commit 258ae1cde4cdb5092edeb00000dc6a50f0af2c1c
Author: cstella <ceste...@gmail.com>
Date: 2016-05-27T02:46:35Z
Added validation framework.
commit d2ae7b3a8a96787523755ce0107c64909e2729bd
Author: cstella <ceste...@gmail.com>
Date: 2016-05-27T03:39:52Z
Updating validators to work with unit tests.
commit ccc42b3d7390e98ade326c5651719d9b6d4533b8
Author: cstella <ceste...@gmail.com>
Date: 2016-05-27T13:50:30Z
Updating readme.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
