[ https://issues.apache.org/jira/browse/METRON-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156273#comment-16156273 ]
ASF GitHub Bot commented on METRON-1162: ---------------------------------------- GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/736 METRON-1162 Apply Live Messages to the Profile Debugger I want to be able to use `PROFILER_APPLY` with live messages from a Metron cluster. For example, I would like to be able to grab 10 messages from my production Metron and then apply them in my debugger. This would be tremendously helpful for iterating between the debugger and a live cluster when troubleshooting. ``` p := PROFILER_INIT(conf) msgs := KAFKA_GET("indexing", 10) PROFILER_APPLY(msgs, p) PROFILER_FLUSH(p) ``` The `PROFILER_APPLY` function does not currently accept a list of messages. It accepts a String that is a JSON list, but not a Stellar list. For `PROFILER_APPLY` to play happy with `KAFKA_GET` (and other potential uses) it needs to accept a List. This PR makes `PROFILER_APPLY` rather flexible in the types of arguments that it accepts. * String containing one JSON message * String containing an array of JSON messages * List containing a JSONObject * List containing a String of one JSON message * List containing a String that is an array of JSON messages ### Testing This can be tested by launching the REPL and running the following. Create a Profiler definition. ``` [Stellar]>>> conf := SHELL_EDIT() [Stellar]>>> conf { "profiles": [ { "profile": "hello-world", "onlyif": "exists(ip_src_addr)", "foreach": "ip_src_addr", "init": { "count": "0" }, "update": { "count": "count + 1" }, "result": "count" } ] } ``` Create a sample message. ``` [Stellar]>>> msg := SHELL_EDIT() [Stellar]>>> msg { "ip_src_addr": "10.0.0.1", "protocol": "HTTPS", "length": "10", "bytes_in": "234" } ``` Initialize the Profiler. ``` [Stellar]>>> p := PROFILER_INIT(conf) [Stellar]>>> p Profiler{1 profile(s), 0 messages(s), 0 route(s)} ``` Apply 3 messages at a time by passing in a list. ``` [Stellar]>>> PROFILER_APPLY([msg, msg, msg], p) Profiler{1 profile(s), 3 messages(s), 3 route(s)} ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1162 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/736.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #736 ---- commit 1cd5794eb7cadbd9923a27d6fe5e63c63e75ccb6 Author: Nick Allen <n...@nickallen.org> Date: 2017-09-06T20:39:01Z METRON-1160 Blueprint configuration validation failed: Missing required properties commit b42f27635f640f8495364af6fc01dedc7c762d94 Author: Nick Allen <n...@nickallen.org> Date: 2017-09-07T00:55:04Z METRON-1162 Apply Live Messages to the Profile Debugger commit 21005351eb000ac0c85fe5a1702b850ec25ca092 Author: Nick Allen <n...@nickallen.org> Date: 2017-09-07T01:03:48Z Revert "METRON-1160 Blueprint configuration validation failed: Missing required properties" This reverts commit 1cd5794eb7cadbd9923a27d6fe5e63c63e75ccb6. ---- > Apply Live Messages to the Profile Debugger > ------------------------------------------- > > Key: METRON-1162 > URL: https://issues.apache.org/jira/browse/METRON-1162 > Project: Metron > Issue Type: Improvement > Affects Versions: 0.4.0 > Reporter: Nick Allen > Assignee: Nick Allen > Fix For: Next + 1 > > > I want to be able to use `PROFILER_APPLY` with live messages from a Metron > cluster. For example, I would like to be able to grab 10 messages from Kafka > and then apply them to my debugger. This would be tremendously helpful for > iterating between the debugger and a live cluster when troubleshooting. > {code} > p := PROFILER_INIT(conf) > msgs := KAFKA_GET("indexing", 10) > PROFILER_APPLY(msgs, p) > PROFILER_FLUSH(p) > {code} > The `PROFILER_APPLY` function does not currently accept a list of messages. > It accepts a String that is a JSON list, but not a Stellar list. For > `PROFILER_APPLY` to play happy with `KAFKA_GET` (and other potential uses) it > needs to accept a List. -- This message was sent by Atlassian JIRA (v6.4.14#64029)