Repository: metron Updated Branches: refs/heads/master 1a4f19c86 -> c73c7af1d
METRON-1132 Enhance Profiler Debug Functions (nickwallen) closes apache/metron#716 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/c73c7af1 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/c73c7af1 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/c73c7af1 Branch: refs/heads/master Commit: c73c7af1dc3cb6502517bc7ac1d37c631ec3dbe5 Parents: 1a4f19c Author: nickwallen <n...@nickallen.org> Authored: Thu Aug 31 10:38:21 2017 -0400 Committer: nickallen <nickal...@apache.org> Committed: Thu Aug 31 10:38:21 2017 -0400 ---------------------------------------------------------------------- .../metron-profiler-client/README.md | 21 +++-- .../client/stellar/ProfilerFunctions.java | 35 ++++++-- .../client/stellar/ProfilerFunctionsTest.java | 93 ++++++++++++++++++-- .../metron/profiler/StandAloneProfiler.java | 40 +++++++++ metron-analytics/metron-profiler/README.md | 2 + 5 files changed, 172 insertions(+), 19 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/c73c7af1/metron-analytics/metron-profiler-client/README.md ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-profiler-client/README.md b/metron-analytics/metron-profiler-client/README.md index a8b5a55..27aa3f4 100644 --- a/metron-analytics/metron-profiler-client/README.md +++ b/metron-analytics/metron-profiler-client/README.md @@ -418,8 +418,11 @@ Follow these steps in the Stellar REPL to see how it can be used to help create ``` [Stellar]>>> profiler := PROFILER_INIT(conf) [Stellar]>>> profiler - org.apache.metron.profiler.StandAloneProfiler@4f8ef473 + Profiler{1 profile(s), 0 messages(s), 0 route(s)} ``` + The profiler itself will show the number of profiles defined, the number of messages applied, and the number of routes taken. + + A route is defined when a message is applied to a specific profile. If a message is applied and not needed by any profile, then there are no routes. If a message is needed by one profile, then one route has been defined. If a message is needed by two profiles, then two routes have been defined. 1. Create a message to simulate the type of telemetry that you expect to be profiled. As an example, in the editor copy/paste the JSON below. ``` @@ -436,14 +439,18 @@ Follow these steps in the Stellar REPL to see how it can be used to help create 1. Apply some telemetry messages to your profiles. The following applies the same message 3 times. ``` [Stellar]>>> PROFILER_APPLY(message, profiler) - org.apache.metron.profiler.StandAloneProfiler@4f8ef473 - + Profiler{1 profile(s), 1 messages(s), 1 route(s)} + ``` + ``` [Stellar]>>> PROFILER_APPLY(message, profiler) - org.apache.metron.profiler.StandAloneProfiler@4f8ef473 - + Profiler{1 profile(s), 2 messages(s), 2 route(s)} + ``` + ``` [Stellar]>>> PROFILER_APPLY(message, profiler) - org.apache.metron.profiler.StandAloneProfiler@4f8ef473 + Profiler{1 profile(s), 3 messages(s), 3 route(s)} ``` + + It is also possible to apply multiple messages at once. This is useful when testing against a larger set of data. To do this, create a string that contains a JSON array of messages and pass that to the `PROFILER_APPLY` function. 1. Flush the Profiler to see what has been calculated. A flush is what occurs at the end of each 15 minute period in the Profiler. The result is a list of profile measurements. Each measurement is a map containing detailed information about the profile data that has been generated. ``` @@ -455,4 +462,4 @@ Follow these steps in the Stellar REPL to see how it can be used to help create This profile simply counts the number of messages by IP source address. Notice that the value is '3' for the entity '10.0.0.1' as we applied 3 messages with an 'ip_src_addr' of '10.0.0.1'. There will always be one measurement for each [profile, entity] pair. -1. If you are unhappy with the data that has been generated, then 'wash, rinse and repeat' this process. After you are satisfied with the data being generated by the profile, then follow the [Getting Started](../metron-profiler#getting-started) guide to use the profile against your live, streaming data in a Metron cluster. \ No newline at end of file +1. If you are unhappy with the data that has been generated, then 'wash, rinse and repeat' this process. Once you are happy with the profile that was created, follow the [Getting Started](../metron-profiler#getting-started) guide to use the profile against your live, streaming data in a Metron cluster. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/metron/blob/c73c7af1/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java b/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java index 827e1c4..8df5ca8 100644 --- a/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java +++ b/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java @@ -28,6 +28,8 @@ import org.apache.metron.stellar.dsl.Context; import org.apache.metron.stellar.dsl.ParseException; import org.apache.metron.stellar.dsl.Stellar; import org.apache.metron.stellar.dsl.StellarFunction; +import org.apache.storm.shade.org.apache.commons.lang.ClassUtils; +import org.json.simple.JSONArray; import org.json.simple.JSONObject; import org.json.simple.parser.JSONParser; import org.slf4j.LoggerFactory; @@ -107,7 +109,7 @@ public class ProfilerFunctions { name="APPLY", description="Apply a message to a local profile runner.", params={ - "message", "The message to apply.", + "message(s)", "The message to apply. A JSON list can be used to apply multiple messages.", "profiler", "A local profile runner returned by PROFILER_INIT." }, returns="The local profile runner." @@ -129,16 +131,33 @@ public class ProfilerFunctions { @Override public Object apply(List<Object> args, Context context) throws ParseException { - // user must provide the json telemetry message + // user must provide the message as a string String arg0 = Util.getArg(0, String.class, args); if(arg0 == null) { throw new IllegalArgumentException(format("expected string, found null")); } - // parse the message - JSONObject message; + // there could be one or more messages + List<JSONObject> messages = new ArrayList<>(); try { - message = (JSONObject) parser.parse(arg0); + Object parsedArg0 = parser.parse(arg0); + if(parsedArg0 instanceof JSONObject) { + // if there is only one message + messages.add((JSONObject) parsedArg0); + + } else if(parsedArg0 instanceof JSONArray) { + // there are multiple messages + JSONArray jsonArray = (JSONArray) parsedArg0; + for(Object json: jsonArray) { + if(json instanceof JSONObject) { + messages.add((JSONObject) json); + + } else { + throw new IllegalArgumentException(format("invalid message: found '%s', expected JSONObject", + ClassUtils.getShortClassName(json, "null"))); + } + } + } } catch(org.json.simple.parser.ParseException e) { throw new IllegalArgumentException("invalid message", e); @@ -147,10 +166,12 @@ public class ProfilerFunctions { // user must provide the stand alone profiler StandAloneProfiler profiler = Util.getArg(1, StandAloneProfiler.class, args); try { - profiler.apply(message); + for(JSONObject message : messages) { + profiler.apply(message); + } } catch(ExecutionException e) { - throw new IllegalArgumentException(e); + throw new IllegalArgumentException(format("Failed to apply message; error=%s", e.getMessage()), e); } return profiler; http://git-wip-us.apache.org/repos/asf/metron/blob/c73c7af1/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/ProfilerFunctionsTest.java ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/ProfilerFunctionsTest.java b/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/ProfilerFunctionsTest.java index 9cc8046..bad3efe 100644 --- a/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/ProfilerFunctionsTest.java +++ b/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/ProfilerFunctionsTest.java @@ -58,6 +58,28 @@ public class ProfilerFunctionsTest { private String message; /** + * [ + * { + * "ip_src_addr": "10.0.0.1", + * "ip_dst_addr": "10.0.0.2", + * "source.type": "test", + * }, + * { + * "ip_src_addr": "10.0.0.1", + * "ip_dst_addr": "10.0.0.2", + * "source.type": "test", + * }, + * { + * "ip_src_addr": "10.0.0.1", + * "ip_dst_addr": "10.0.0.2", + * "source.type": "test", + * } + * ] + */ + @Multiline + private String messages; + + /** * { * "profiles": [ * { @@ -108,7 +130,9 @@ public class ProfilerFunctionsTest { state.put("config", "{ \"profiles\" : [] }"); StandAloneProfiler profiler = run("PROFILER_INIT(config)", StandAloneProfiler.class); assertNotNull(profiler); - assertEquals(0, profiler.getConfig().getProfiles().size()); + assertEquals(0, profiler.getProfileCount()); + assertEquals(0, profiler.getMessageCount()); + assertEquals(0, profiler.getRouteCount()); } @Test @@ -116,7 +140,9 @@ public class ProfilerFunctionsTest { state.put("config", helloWorldProfilerDef); StandAloneProfiler profiler = run("PROFILER_INIT(config)", StandAloneProfiler.class); assertNotNull(profiler); - assertEquals(1, profiler.getConfig().getProfiles().size()); + assertEquals(1, profiler.getProfileCount()); + assertEquals(0, profiler.getMessageCount()); + assertEquals(0, profiler.getRouteCount()); } @Test(expected = IllegalArgumentException.class) @@ -144,7 +170,9 @@ public class ProfilerFunctionsTest { StandAloneProfiler profiler = executor.execute(expression, state, StandAloneProfiler.class); assertNotNull(profiler); - assertEquals(1, profiler.getConfig().getProfiles().size()); + assertEquals(1, profiler.getProfileCount()); + assertEquals(0, profiler.getMessageCount()); + assertEquals(0, profiler.getRouteCount()); } @Test @@ -158,19 +186,74 @@ public class ProfilerFunctionsTest { // apply a message to the profiler state.put("message", message); StandAloneProfiler result = run("PROFILER_APPLY(message, profiler)", StandAloneProfiler.class); + + // validate + assertSame(profiler, result); + assertEquals(1, profiler.getProfileCount()); + assertEquals(1, profiler.getMessageCount()); + assertEquals(1, profiler.getRouteCount()); + } + + @Test + public void testProfilerApplyWithMultipleMessages() { + + // initialize the profiler + state.put("config", helloWorldProfilerDef); + StandAloneProfiler profiler = run("PROFILER_INIT(config)", StandAloneProfiler.class); + state.put("profiler", profiler); + + // apply a message to the profiler + state.put("messages", messages); + StandAloneProfiler result = run("PROFILER_APPLY(messages, profiler)", StandAloneProfiler.class); + + // validate assertSame(profiler, result); + assertEquals(1, profiler.getProfileCount()); + assertEquals(3, profiler.getMessageCount()); + assertEquals(3, profiler.getRouteCount()); + } + + @Test + public void testProfilerApplyWithEmptyList() { + + // initialize the profiler + state.put("config", helloWorldProfilerDef); + StandAloneProfiler profiler = run("PROFILER_INIT(config)", StandAloneProfiler.class); + state.put("profiler", profiler); + + // apply a message to the profiler + state.put("messages", "[ ]"); + StandAloneProfiler result = run("PROFILER_APPLY(messages, profiler)", StandAloneProfiler.class); + + // validate + assertSame(profiler, result); + assertEquals(1, profiler.getProfileCount()); + assertEquals(0, profiler.getMessageCount()); + assertEquals(0, profiler.getRouteCount()); } @Test(expected = IllegalArgumentException.class) - public void testProfilerApplyNoArgs() { + public void testProfilerApplyWithNoArgs() { run("PROFILER_APPLY()", StandAloneProfiler.class); } @Test(expected = IllegalArgumentException.class) - public void testProfilerApplyInvalidArg() { + public void testProfilerApplyWithInvalidArg() { run("PROFILER_APPLY(undefined)", StandAloneProfiler.class); } + @Test(expected = IllegalArgumentException.class) + public void testProfilerApplyWithNullMessage() { + + // initialize the profiler + state.put("config", helloWorldProfilerDef); + StandAloneProfiler profiler = run("PROFILER_INIT(config)", StandAloneProfiler.class); + state.put("profiler", profiler); + + // there is no 'messages' variable + StandAloneProfiler result = run("PROFILER_APPLY(messages, profiler)", StandAloneProfiler.class); + } + @Test public void testProfilerFlush() { http://git-wip-us.apache.org/repos/asf/metron/blob/c73c7af1/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/StandAloneProfiler.java ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/StandAloneProfiler.java b/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/StandAloneProfiler.java index cf034c8..6db7079 100644 --- a/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/StandAloneProfiler.java +++ b/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/StandAloneProfiler.java @@ -53,12 +53,28 @@ public class StandAloneProfiler { */ private MessageDistributor distributor; + /** + * Counts the number of messages that have been applied. + */ + private int messageCount; + + /** + * Counts the number of routes. + * + * If a message is not needed by any profiles, then there are 0 routes. + * If a message is needed by 1 profile then there is 1 route. + * If a message is needed by 2 profiles then there are 2 routes. + */ + private int routeCount; + public StandAloneProfiler(ProfilerConfig config, long periodDurationMillis, Context context) { this.context = context; this.config = config; this.router = new DefaultMessageRouter(context); // the period TTL does not matter in this context this.distributor = new DefaultMessageDistributor(periodDurationMillis, Long.MAX_VALUE); + this.messageCount = 0; + this.routeCount = 0; } /** @@ -72,6 +88,18 @@ public class StandAloneProfiler { for(MessageRoute route : routes) { distributor.distribute(message, route, context); } + + routeCount += routes.size(); + messageCount += 1; + } + + @Override + public String toString() { + return "Profiler{" + + getProfileCount() + " profile(s), " + + getMessageCount() + " messages(s), " + + getRouteCount() + " route(s)" + + '}'; } /** @@ -85,4 +113,16 @@ public class StandAloneProfiler { public ProfilerConfig getConfig() { return config; } + + public int getProfileCount() { + return (config == null) ? 0: config.getProfiles().size(); + } + + public int getMessageCount() { + return messageCount; + } + + public int getRouteCount() { + return routeCount; + } } http://git-wip-us.apache.org/repos/asf/metron/blob/c73c7af1/metron-analytics/metron-profiler/README.md ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-profiler/README.md b/metron-analytics/metron-profiler/README.md index f27af59..b7aed55 100644 --- a/metron-analytics/metron-profiler/README.md +++ b/metron-analytics/metron-profiler/README.md @@ -281,6 +281,8 @@ In the following example, three values, the minimum, the maximum and the mean ar A numeric value that defines how many days the profile data is retained. After this time, the data expires and is no longer accessible. If no value is defined, the data does not expire. +The REPL can be a powerful for developing profiles. Read all about [Developing Profiles](../metron-profiler-client/#developing_profiles). + ## Configuring the Profiler The Profiler runs as an independent Storm topology. The configuration for the Profiler topology is stored in local filesystem at `$METRON_HOME/config/profiler.properties`.
