GitHub user merrimanr opened a pull request:
https://github.com/apache/incubator-metron/pull/308
Grok patterns are now read from zookeeper parser config property
"grokPattern"
This PR moves Grok patterns from HDFS to a property in Zookeeper
("SensorParserConfig.parserConfig.grokPattern"). Most of the important changes
were made to the GrokParser class and related tests. There were 2 primary
challenges with implementing this:
- A change to a Grok pattern in Zookeeper should take effect immediately
without a topology restart. Currently the MessageParser does not have access
to the Zookeeper config.
- Grok patterns must now comply with JSON formatting standards (files had
less restrictions)
The first challenge was solved by changing the MessageParser.parse method
to include an additional "SensorParserConfig" parameter that the GrokParser can
use to determine if the grok pattern has changed and reinit if necessary. The
ParserBolt now passes in the most recent SensorParserConfig to the
MessageParser for every message processed. This is similar to how the writer
interfaces work. I believe this approach is simpler than having the ParserBolt
listen for changes and reconfigure/reinit the GrokParser. Changing this
interface makes this PR look much bigger than it actually is because all the
parsers and parser tests had to be adjusted. Most of files included are 1 line
changes.
The second challenge was overcome by using '\n' to separate lines inside
the grok pattern JSON property. An extra '\' also had to be added to '\'
escape characters in the grok pattern files. The advantage of this is that no
special serialization/deserialization is needed inside of the GrokParser code.
The disadvantage is that the patterns are not as human-readable.
All unit tests and integration tests are passing. I added an extra unit
test in SampleGrokParserTest to simulate and test a config being updated. I
also tested yaf and squid sensors in quick-dev by updating the grokPattern
properties and verifying the new patterns were applied.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-498
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/308.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #308
----
commit 56be7560dd42fefadfb0d88691b76993943da449
Author: rmerriman <[email protected]>
Date: 2016-10-12T18:46:45Z
Grok patterns are now read from zookeeper parser config property
"grokPattern"
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---