The changes are pretty simple (pre-compile the grok, duh). Most other grok parser just use a single expression, which is already pre-compiled (/checks assumption in code) so really it’s just the ASA one because of it’s strange two stage grok.
Shame, it would have been nice to find some more low hanging fruit. Simon > On 9 Jun 2017, at 01:52, Otto Fowler <[email protected]> wrote: > > Are these changes that all grok parsers can benefit from? Are your changes > to the base classes that they use or asa only? > > > > On June 8, 2017 at 20:49:49, Simon Elliston Ball ([email protected] > <mailto:[email protected]>) wrote: > >> I got mildly interested in parser performance as a result of some recent >> work on tuning, and did some very quick benchmarking with Predfix on the ASA >> parser (which I hadn’t really cared about enough due to relatively low >> volume previously). >> >> That said, it’s not exactly perf optimised. 3 runs of 1000 iterations on my >> laptop as a micro-benchmark in Predfix (I know, scientific, right), with >> some changes (basically pushing all the grok statements up to pre-compile in >> init (the parser currently uses one grok to do the syslog bit and figure out >> which grok it needs for the second half, so this makes for a large number of >> Grok objects upfront, which I think we can live with. >> >> Do you think we should do this benchmarking properly, and extend? Anyone >> have thoughts about how to build parser benchmarks in to our test suite >> properly? >> >> Also, since these are showing approx 20 times improvement on the P95 >> interval, do we think it’s worth the memory (not measured, but 39 Grok >> objects hanging around? If so I’ll get it JIRAed up and push my new version. >> >> Run results:- >> >> Base line (current master as is) >> |= Benchmark >> ==================================================================================| >> >> | - | unit | sum | min | max | avg | stddev | conf95 | runs | >> |========================================= TimeMeter >> ==========================================| >> |. AsaBenchmark >> ...............................................................................| >> >> | parserBenchmark | ms | 5597.98 | 04.90 | 159.02 | 05.60 | 04.89 | >> [05.01-06.20] | 1000.00 | >> | parserBenchmark | ms | 5503.91 | 04.82 | 149.60 | 05.50 | 04.59 | >> [05.00-05.90] | 1000.00 | >> | parserBenchmark | ms | 5620.90 | 04.80 | 152.83 | 05.62 | 04.71 | >> [04.98-06.73] | 1000.00 | >> |==============================================================================================| >> >> >> Syslog element of Grok pulled out and pre-compiled >> >> |= Benchmark >> ==================================================================================| >> >> | - | unit | sum | min | max | avg | stddev | conf95 | runs | >> |========================================= TimeMeter >> ==========================================| >> |. AsaBenchmark >> ...............................................................................| >> >> | parserBenchmark | ms | 4299.91 | 03.29 | 120.06 | 04.30 | 03.89 | >> [03.36-07.10] | 1000.00 | >> | parserBenchmark | ms | 4206.98 | 03.31 | 129.41 | 04.21 | 04.07 | >> [03.46-05.44] | 1000.00 | >> | parserBenchmark | ms | 3843.05 | 03.28 | 119.39 | 03.84 | 03.79 | >> [03.33-04.55] | 1000.00 | >> |==============================================================================================| >> >> >> With all precompiled in a hash map (more memory use, but not by a lot) >> >> |= Benchmark >> =================================================================================| >> >> | - | unit | sum | min | max | avg | stddev | conf95 | runs | >> |========================================= TimeMeter >> =========================================| >> |. AsaBenchmark >> ..............................................................................| >> >> | parserBenchmark | ms | 514.68 | 00.22 | 112.35 | 00.51 | 03.55 | >> [00.24-00.79] | 1000.00 | >> | parserBenchmark | ms | 472.42 | 00.22 | 105.19 | 00.47 | 03.32 | >> [00.23-00.70] | 1000.00 | >> | parserBenchmark | ms | 484.40 | 00.21 | 103.71 | 00.48 | 03.27 | >> [00.24-00.76] | 1000.00 | >> |==============================================================================================| >> >> >> Simon
