For some reason my overall comment on this PR failed to be copied to the list:
This is a really nice new feature. It works and is clean. It rates a +1. But I think we are likely to be processing thousands or perhaps millions of data points at any given time, and so the constant re-parsing of the bounds list is troublesome. Also for the STATS_BIN function, the percentile bounds list should be transformed once into a value bounds list, rather than applying the boundFunc many times to (on average) half the bounds in the list, per input value. Unlike the Profiler, there is no config caching between calls to the Stellar Function, because this "configuration" is done every time, in-line, rather than in ZK. But we don't want all the complexity of ZK just for this little binning function. And we want multiple different binning functions to be in use at the same time, without needing complex scope management. One solution would be to treat it like Python does with regex, and provide a compiler function. What if we have COMPILE_BIN(bounds) and COMPILE_STATS_BIN(stats, bounds) Each would return an opaque key (or an integer) that references a cached pre-parsed setup; it can be thread-safe as it would be read-only. Then we would invoke with BIN(key, value) and STATS_BIN(key, value) Normally I would say this is an optimization and we should leave it for later. But then we would be stuck with the inefficient, non-compiled form of the BIN and STATS_BIN functions. Your call, @cestella . I don't want to get in the way of progress, I just feel obligated to bring it up. --Matt On 12/21/16, 11:50 PM, "mattf-horton" <[email protected]> wrote: Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/401#discussion_r93574334 --- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java --- @@ -425,4 +428,61 @@ public Object apply(List<Object> args) { return result; } } + + /** + * Calculates the statistical bin that a value falls in. + */ + @Stellar(namespace = "STATS", name = "BIN" + , description = "Computes the bin that the value is in based on the statistical distribution." + , params = { + "stats - The Stellar statistics object" + , "value - The value to bin" + , "bounds? - A list of percentile bin bounds (excluding min and max) or a string representing a known and common set of bins. " + + "For convenience, we have provided QUARTILE, QUINTILE, and DECILE which you can pass in as a string arg." + + " If this argument is omitted, then we assume a Quartile bin split." + } + ,returns = "Which bin N the value falls in such that bound(N-1) < value <= bound(N). " + + "No min and max bounds are provided, so values smaller than the 0'th bound go in the 0'th bin, " + + "and values greater than the last bound go in the M'th bin." + ) + public static class StatsBin extends BaseStellarFunction { + public enum BinSplits { + QUARTILE(ImmutableList.of(25.0, 50.0, 75.0)), + QUINTILE(ImmutableList.of(20.0, 40.0, 60.0, 80.0)), + DECILE(ImmutableList.of(10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0)) + ; + public final List<Double> split; + BinSplits(List<Double> split) { + this.split = split; + } + + public static List<Double> getSplit(Object o) { + if(o instanceof String) { + return BinSplits.valueOf((String)o).split; + } + else if(o instanceof List) { + List<Double> ret = new ArrayList<>(); + for(Object valO : (List<Object>)o) { + ret.add(ConversionUtils.convert(valO, Double.class)); --- End diff -- During this step we need to validate that the bounds list is strictly increasing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [email protected] or file a JIRA ticket with INFRA. ---
