[ https://issues.apache.org/jira/browse/METRON-1925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711892#comment-16711892 ]
ASF GitHub Bot commented on METRON-1925: ---------------------------------------- GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1292 METRON-1925 Provide Verbose View of Profile Results in REPL ## Motivation When viewing profile measurements in the REPL using PROFILE_GET, you simply get back a list of values. It is not easy to determine from which time period the measurements were taken. For example, are the following values all sequential? Are there any gaps in the measurements taken over the past 30 minutes? When was the first measurement taken? ``` [Stellar]>>> PROFILE_GET("hello-world","global",PROFILE_FIXED(30, "MINUTES")) [2655, 1170, 1185, 1170, 1185, 1215, 1200, 1170] ``` The `PROFILE_GET` function was designed to return values that serve as input to other functions. It was not designed to return values in a human-readable form that can be easily understood. We need another way to query for profile measurements in the REPL that provides a user with a better understanding of the profile measurements. ## Solution This PR provides a new function called `PROFILE_VIEW`. It is effectively a "verbose mode" for `PROFILE_GET`. For lack of a better name, I just called it `PROFILE_VIEW`. I would be open to alternatives. I did not want to add additional options to the already complex `PROFILE_GET`. * Description: Retrieves a series of measurements from a stored profile. Provides a more verbose view of each measurement than PROFILE_GET. Returns a map containing the profile name, entity, period id, period start, period end for each profile measurement. * Arguments: profile - The name of the profile. entity - The name of the entity. periods - The list of profile periods to fetch. Use PROFILE_WINDOW or PROFILE_FIXED. groups - Optional, The groups to retrieve. Must correspond to the 'groupBy' list used during profile creation. Defaults to an empty list, meaning no groups. * Returns: A map for each profile measurement containing the profile name, entity, period, and value. ## Test Drive 1. Spin-up Full Dev and create a profile. Follow the Profiler README. Reduce the profile period if you are impatient. 1. Open up the REPL and retrieve the values using `PROFILE_GET`. Notice that I have no idea when the first measurement was taken, if the values are sequential, if there are gaps in the values and how big. ``` [Stellar]>>> PROFILE_GET("hello-world","global",PROFILE_FIXED(30, "MINUTES")) [1185, 1170, 1185, 1215, 1200, 1170, 5425, 1155, 1215, 1200] ``` 1. Now use `PROFILE_VIEW` to retrieve the same results. ``` [Stellar]>>> results := PROFILE_VIEW("hello-world","global",PROFILE_FIXED(30, "MINUTES")) [{period.start=1544119560000, period=12867663, profile=hello-world, period.end=1544119680000, groups=[], value=1185, entity=global}, {period.start=1544119680000, period=12867664, profile=hello-world, period.end=1544119800000, groups=[], value=1170, entity=global}, {period.start=1544119800000, period=12867665, profile=hello-world, period.end=1544119920000, groups=[], value=1185, entity=global}, {period.start=1544119920000, period=12867666, profile=hello-world, period.end=1544120040000, groups=[], value=1215, entity=global}, {period.start=1544120040000, period=12867667, profile=hello-world, period.end=1544120160000, groups=[], value=1200, entity=global}, {period.start=1544120160000, period=12867668, profile=hello-world, period.end=1544120280000, groups=[], value=1170, entity=global}, {period.start=1544120880000, period=12867674, profile=hello-world, period.end=1544121000000, groups=[], value=5425, entity=global}, {period.start=1544121000000, period=12867675, profile=hello-world, period.end=1544121120000, groups=[], value=1155, entity=global}, {period.start=1544121120000, period=12867676, profile=hello-world, period.end=1544121240000, groups=[], value=1215, entity=global}, {period.start=1544121240000, period=12867677, profile=hello-world, period.end=1544121360000, groups=[], value=1200, entity=global}] ``` 1. For each measurement, I have a map containing the period, period start, period end, profile name, entity, groups, and value. With this I can better answer some of the questions above. ``` { profile=hello-world, entity=global, period=12867663, period.start=1544119560000, period.end=1544119680000, groups=[], value=1185 } ``` 1. When was the first measurement taken? ``` [Stellar]>>> GET(results, 0) {period.start=1544119560000, period=12867663, profile=hello-world, period.end=1544119680000, groups=[], value=1185, entity=global} ``` I can see that the first period started at 1544119560000 or Thu Dec 06 2018 18:06:00 UTC. 1. Are these measurements sequential? Are there any gaps? I can use `MAP` on the list of results to extract just the periods from each profile measurement. The period is a monotonically increasing value. ``` [Stellar]>>> MAP(results, m -> MAP_GET("period", m)) [12867663, 12867664, 12867665, 12867666, 12867667, 12867668, 12867674, 12867675, 12867676, 12867677] ``` From this I can tell that there is a gap in the measurements. * The first period here is 12867663 and the last is 12867677. * I can see that the first 6 measurements are sequential up to period 12867668. * Then there is a gap of 5 periods (12867669 - 12867673) before the measurements resume at period 12867674. 1. How big was that gap? We know that we missed 5 periods, but how big is that gap? Here is when each period starts in epoch milliseconds. ``` [Stellar]>>> MAP(results, m -> MAP_GET("period.start", m)) [1544119560000, 1544119680000, 1544119800000, 1544119920000, 1544120040000, 1544120160000, 1544120880000, 1544121000000, 1544121120000, 1544121240000] ``` I can find the difference between the 5th and 6th period. The different is about 720000 milliseconds, which is 12 minutes. ``` [Stellar]>>> MAP_GET("period.start", GET(results, 5)) 1544120160000 [Stellar]>>> MAP_GET("period.start", GET(results, 6)) 1544120880000 [Stellar]>>> MAP_GET("period.start", GET(results, 6)) - MAP_GET("period.start", GET(results, 5)) 720000 ``` ## Changes * Altered the ProfilerClient so that it returns the `ProfileMeasurement` values instead of just a list of the raw values stored in HBase. * Removed one of the methods of ProfileClient that was not being used to simplify things. * Updated `PROFILE_GET` to work with the altered ProfilerClient. * Added `PROFILE_VIEW`. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1925 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1292.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1292 ---- commit 8a058e6274d712ed046325ccb408063f8ac9edbf Author: Nick Allen <nick@...> Date: 2018-12-04T13:54:45Z METRON-1925 Provide Verbose View of Profile Results in REPL ---- > Provide Verbose View of Profile Results in REPL > ----------------------------------------------- > > Key: METRON-1925 > URL: https://issues.apache.org/jira/browse/METRON-1925 > Project: Metron > Issue Type: Bug > Reporter: Nick Allen > Assignee: Nick Allen > Priority: Major > > When viewing profile measurements in the REPL using PROFILE_GET, you simply > get back a list of values. It is not easy to determine from which time period > the measurements were taken. > For example, are the following values all sequential? Are there any gaps in > the measurements taken over the past 30 minutes? When was the first > measurement taken? > {code:java} > [Stellar]>>> PROFILE_GET("hello-world","global",PROFILE_FIXED(30, "MINUTES")) > [2655, 1170, 1185, 1170, 1185, 1215, 1200, 1170]{code} > The `PROFILE_GET` function was designed to return values that serve as input > to other functions. It was not designed to return values in a human-readable > form that can be easily understood. > We need another way to query for profile measurements in the REPL that gives > me a better understanding of the profile measurements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)