GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/736
METRON-1162 Apply Live Messages to the Profile Debugger
I want to be able to use `PROFILER_APPLY` with live messages from a Metron
cluster. For example, I would like to be able to grab 10 messages from my
production Metron and then apply them in my debugger. This would be
tremendously helpful for iterating between the debugger and a live cluster when
troubleshooting.
```
p := PROFILER_INIT(conf)
msgs := KAFKA_GET("indexing", 10)
PROFILER_APPLY(msgs, p)
PROFILER_FLUSH(p)
```
The `PROFILER_APPLY` function does not currently accept a list of messages.
It accepts a String that is a JSON list, but not a Stellar list. For
`PROFILER_APPLY` to play happy with `KAFKA_GET` (and other potential uses) it
needs to accept a List.
This PR makes `PROFILER_APPLY` rather flexible in the types of arguments
that it accepts.
* String containing one JSON message
* String containing an array of JSON messages
* List containing a JSONObject
* List containing a String of one JSON message
* List containing a String that is an array of JSON messages
### Testing
This can be tested by launching the REPL and running the following.
Create a Profiler definition.
```
[Stellar]>>> conf := SHELL_EDIT()
[Stellar]>>> conf
{
"profiles": [
{
"profile": "hello-world",
"onlyif": "exists(ip_src_addr)",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
```
Create a sample message.
```
[Stellar]>>> msg := SHELL_EDIT()
[Stellar]>>> msg
{
"ip_src_addr": "10.0.0.1",
"protocol": "HTTPS",
"length": "10",
"bytes_in": "234"
}
```
Initialize the Profiler.
```
[Stellar]>>> p := PROFILER_INIT(conf)
[Stellar]>>> p
Profiler{1 profile(s), 0 messages(s), 0 route(s)}
```
Apply 3 messages at a time by passing in a list.
```
[Stellar]>>> PROFILER_APPLY([msg, msg, msg], p)
Profiler{1 profile(s), 3 messages(s), 3 route(s)}
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1162
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/736.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #736
----
commit 1cd5794eb7cadbd9923a27d6fe5e63c63e75ccb6
Author: Nick Allen <[email protected]>
Date: 2017-09-06T20:39:01Z
METRON-1160 Blueprint configuration validation failed: Missing required
properties
commit b42f27635f640f8495364af6fc01dedc7c762d94
Author: Nick Allen <[email protected]>
Date: 2017-09-07T00:55:04Z
METRON-1162 Apply Live Messages to the Profile Debugger
commit 21005351eb000ac0c85fe5a1702b850ec25ca092
Author: Nick Allen <[email protected]>
Date: 2017-09-07T01:03:48Z
Revert "METRON-1160 Blueprint configuration validation failed: Missing
required properties"
This reverts commit 1cd5794eb7cadbd9923a27d6fe5e63c63e75ccb6.
----
---
