GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/621
METRON-1001: Allow metron to ingest parser metadata along with data
## Contributor Comments
Currently, we only ingest data in Metron. Often, there is valuable metadata
constructed up-stream of Metron that is relevant to enrichment and cross-cuts
many data formats. Take, for instance, a multi-tenancy case where multiple
sources come in and you'd like to tag the data with the customer ID. In this
case you're stuck finding ways to add the metadata to each data source's
format. Rather than do that, we should allow metadata to be ingested along with
the data associated with it.
There are two sources of metadata relevant to support:
* User defined metadata (e.g. customer IDs)
* Environmental metadata (e.g. the kafka topic)
In this PR, metadata can be expressed and passed into metron in two manners:
* As a JSON Map in the key of the message (optional)
* The kafka topic written.
Metadata is exposed in two ways:
* As stellar variables for Stellar field transformations
* Optionally, automatically merged with the message
Two properties were added to the parser config:
* `mergeMetadata` - default false. If true, then metadata is merged with
the message.
* `readMetadata` - default false. If true, then user and environmental
metadata are attempted to be read.
Note: This is backwards compatible, so by default, unless specified, your
parser topologies will function as before.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron METRON-1001
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/621.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #621
----
commit a4e85398e28f926682396d4828b7a6fdfbeefb33
Author: cstella <[email protected]>
Date: 2017-06-21T19:59:52Z
METRON-1001: Allow metron to ingest parser metadata along with data
commit 26b0422d16c1e330bc2a5e39d7ae80b0837ac3a3
Author: cstella <[email protected]>
Date: 2017-06-21T20:10:35Z
turned off ALL metadata reading unless specified.
commit 2ab442757696bf60db82cab43c9790de2eb0def9
Author: cstella <[email protected]>
Date: 2017-06-21T22:04:28Z
Avoid partitioners that use subscribe rather than assign.
commit 0115d877a66c09e33de75cecddb999bbbdc3be8f
Author: cstella <[email protected]>
Date: 2017-06-21T22:11:28Z
Boo, bad storm, partitioners should be serializable.
commit 30830850e79934399508063dbd9b6d9c02524cfa
Author: cstella <[email protected]>
Date: 2017-06-21T22:16:15Z
Reverted to non-manual. That deserves a separate PR.
commit 5b840a8df11f8e04571752f6e0b2fe9d5abfab0a
Author: cstella <[email protected]>
Date: 2017-06-21T22:34:09Z
Updating test.
commit 281779b4ab4b1d663e8bcf4a20194fd3ecf226e2
Author: cstella <[email protected]>
Date: 2017-06-22T14:13:09Z
Reacting to Otto's comments: generalized from being kafka-specific
commit 5e383099e99c5db6a1a7cc907e52f45da812ab61
Author: cstella <[email protected]>
Date: 2017-06-23T00:23:22Z
Moved constants out.
commit ed03677dfcd4cc3616653aefc1645e8a5414c56d
Author: cstella <[email protected]>
Date: 2017-06-23T01:03:54Z
avoid NPE if possible. heh
commit 6ebfcc05cac0a41d22d06b633007f46012201d25
Author: cstella <[email protected]>
Date: 2017-06-23T01:35:47Z
Updated readme
commit 902805b3ea78be0c362f6167dde730453d499455
Author: cstella <[email protected]>
Date: 2017-06-23T13:23:47Z
Killing with slightly more prejudice than otherwise.
commit e04c1ff142ec81b023a0ce9b3b8998521e12a306
Author: cstella <[email protected]>
Date: 2017-06-23T19:36:16Z
Trying to avoid the skadoo of death.
commit 0268d7e333c0595ce3b357c5263b3715b0761827
Author: cstella <[email protected]>
Date: 2017-06-23T20:00:12Z
The spout is too sensitive.
commit 86e56ca2ebddf499c26a40d9f674a0513317bc74
Author: cstella <[email protected]>
Date: 2017-06-26T11:33:30Z
Cleaning up better.
commit 507bfa317dc4007ba0ba51bf0ec8bf9addb38be3
Author: cstella <[email protected]>
Date: 2017-06-26T11:35:53Z
Too aggressive.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
