GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/1084
METRON-1644: Support parser chaining ## Contributor Comments Currently we have only one layer of parsing prior to enrichment, but often real data is more complex. For instance, often data is wrapped in an envelope (e.g. syslog data which contains a field which needs to be parsed). This Jira allows us to support a DAG of parsers prior to enrichment by allowing us to provide a strategy for interpreting what is data and what is metadata in the parser bolt. This enables upstream parsers to pass in a JSON Blob which contains the metadata and have the parser bolt choose which field is the data to be parsed, the remaining fields would be considered metadata (and merged into the resulting data or not depending on our existing rules for handling metadata). To illustrate this better, I've provided a use-case with an example. Note, this PR depends on METRON-1643 and METRON-1642, so those should be reviewed prior to this. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` #### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron enveloped_message_supplier Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1084.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1084 ---- commit d1cb318961277da732650152c8526054769d830b Author: cstella <cestella@...> Date: 2018-06-05T21:07:13Z Adding message supplier. Tests pending. commit ffad876b377f6cf94213b783a8c139e0973c2e00 Author: cstella <cestella@...> Date: 2018-06-06T14:12:55Z Fixed tests commit b349bf0b8fa251846d77cabc8b6aa019154a6425 Author: cstella <cestella@...> Date: 2018-06-06T14:22:53Z missing license commit 71959c89816ddf4d84fef9585cfb03bd7f624612 Author: cstella <cestella@...> Date: 2018-06-07T16:05:14Z Renamed and refactored metadata utilities commit 6e809e4cf9c34c9207d2d7c501db1dc0390c1d0b Author: cstella <cestella@...> Date: 2018-06-07T16:06:51Z compilation issues commit 5c9dd7caa68820d1ca06d99ff3ef2ac7eace7ba6 Author: cstella <cestella@...> Date: 2018-06-07T20:55:10Z Added unit and integration tests commit bbd922e440e3dcfd22d54e76710d75d3fd2f5ab9 Author: cstella <cestella@...> Date: 2018-06-07T22:10:00Z Updating documentation and refactoring. commit b2d1ca8b685306aafe07a65811f8223b6136bf9a Author: cstella <cestella@...> Date: 2018-06-08T02:36:49Z Updated integration test. commit 3b01987a64030a1dd3cccac2d70d7e5d375bac1c Author: cstella <cestella@...> Date: 2018-06-08T03:07:04Z Merge branch 'master' into enveloped_message_supplier commit f940234aec3e3360e12b4015c9cdc9cf01df681d Author: cstella <cestella@...> Date: 2018-06-08T03:30:07Z Fixed test and subtle bug. commit 92e92fa658d5aaeeb92b970449ac7dded0ed675e Author: cstella <cestella@...> Date: 2018-06-08T05:06:53Z typos commit 8c7112095ceddf5b7a1e812b905721991abd91ee Author: cstella <cestella@...> Date: 2018-06-08T05:11:02Z typo commit b9036f8f6feebbb19efb964a66f5ff8131fcbe8d Author: cstella <cestella@...> Date: 2018-06-08T05:21:28Z missed sensorname commit 532add3d533bd73ab7228a09e0dfc50255242c33 Author: cstella <cestella@...> Date: 2018-06-08T06:21:44Z updating grok commit b5140a22fda7fd39f18b5a13c91a95785f3b6a4b Author: cstella <cestella@...> Date: 2018-06-08T06:33:38Z Updating grok commit cfcbcff3353a6773c18cebfa877e350a85e38859 Author: cstella <cestella@...> Date: 2018-06-27T14:19:55Z Merge branch 'master' into enveloped_message_supplier ---- ---