GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1146
METRON-1704 Message Timestamp Logic Should be Shared
The Profiler can operate using either processing time or event time. This
is controlled by the user by defining the "timestampField" option in their
Profiler configuration.
There is logic that extracts the timestamp from a message. If the Profiler
is using processing time, then system time is returned by this logic. If the
Profiler is using event time, then the timestamp is extracted from a message
field.
This logic is currently duplicated across both ports of the Profiler; the
REPL and Storm. This should be pulled into `metron-profiler-common/` so that
the logic can be shared and also used by the Spark port.
This is a pull request against the `METRON-1699-create-batch-profiler`
feature branch.
## Testing
1. Launch the development environment.
1. Ensure alerts are created in the Alerts UI.
1. Ensure the Service Check in Ambari passes.
1. Follow the instructions in the Profiler README to create a basic profile
in the REPL.
1. Follow the instructions in the Profiler README to create a simple
profile using the Profiler topology in Storm.
## Pull Request Checklist
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1704-v2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1146.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1146
----
commit 56c35d4cb2d97c4c34715f4b717f218cc9c8b650
Author: Nick Allen <nick@...>
Date: 2018-07-10T19:42:19Z
Moving message timestamp determination to the MessageRouter. Logic needed
by both Storm and Spark
----
---
