Guangyuan Feng created KYLIN-5561: ------------------------------------- Summary: Optimize the build performance for models containing semi-additive measure Key: KYLIN-5561 URL: https://issues.apache.org/jira/browse/KYLIN-5561 Project: Kylin Issue Type: Bug Components: Modeling Affects Versions: 5.0-alpha Reporter: Guangyuan Feng Assignee: Yaguang Jia Fix For: 5.0-alpha
When building a model with aggregate function `sum_lc`, it takes too much time to complete the calculation even on a small dataset. After dug into it's implementation, we found the root cause is that the `serialize` will always allocate a new array with `1024 * 1024` bytes as the temporary place to store the serialized value of `SumLCCounter`. Actually, only a decimal and a long value of a `SumLCCounter` object should be serialized, generally the serialized data size is about `8 + 8` bytes in 64-bit platform, so obviously the temporary array is too big to store the result. After deduce the init size of the temporary array, for example 32-Bytes, the total time to complete the calculation of `sum_lc` on 10GB datasets, have been reduced from 16min => 4min. Here is the benchmark tests: {code:java} // After optimized # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 10 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: io.kyligence.pe.JmhSumLCApplication.dynamicLength # Run progress: 0.00% complete, ETA 00:04:00 # Fork: 1 of 2 # Warmup Iteration 1: 39082.864 ops/ms Iteration 1: 41760.550 ops/ms Iteration 2: 47911.634 ops/ms Iteration 3: 47353.936 ops/ms Iteration 4: 46888.688 ops/ms Iteration 5: 48378.075 ops/ms # Run progress: 25.00% complete, ETA 00:03:02 # Fork: 2 of 2 # Warmup Iteration 1: 39479.279 ops/ms Iteration 1: 42066.415 ops/ms Iteration 2: 48499.974 ops/ms Iteration 3: 48524.844 ops/ms Iteration 4: 48431.830 ops/ms Iteration 5: 48451.256 ops/ms Result "io.kyligence.pe.JmhSumLCApplication.dynamicLength": 46826.720 ±(99.9%) 4002.887 ops/ms [Average] (min, avg, max) = (41760.550, 46826.720, 48524.844), stdev = 2647.662 CI (99.9%): [42823.833, 50829.607] (assumes normal distribution) // Before optimized # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 10 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: io.kyligence.pe.JmhSumLCApplication.fixLength # Run progress: 50.00% complete, ETA 00:02:01 # Fork: 1 of 2 # Warmup Iteration 1: 22.364 ops/ms Iteration 1: 25.354 ops/ms Iteration 2: 25.252 ops/ms Iteration 3: 20.566 ops/ms Iteration 4: 20.668 ops/ms Iteration 5: 21.585 ops/ms # Run progress: 75.00% complete, ETA 00:01:00 # Fork: 2 of 2 # Warmup Iteration 1: 22.953 ops/ms Iteration 1: 25.362 ops/ms Iteration 2: 24.041 ops/ms Iteration 3: 21.774 ops/ms Iteration 4: 25.131 ops/ms Iteration 5: 25.594 ops/ms Result "io.kyligence.pe.JmhSumLCApplication.fixLength": 23.533 ±(99.9%) 3.210 ops/ms [Average] (min, avg, max) = (20.566, 23.533, 25.594), stdev = 2.123 CI (99.9%): [20.323, 26.743] (assumes normal distribution) # Run complete. Total time: 00:04:03 REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial experiments, perform baseline and negative tests that provide experimental control, make sure the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts. Do not assume the numbers tell you what you want them to tell. Benchmark Mode Cnt Score Error Units JmhSumLCApplication.dynamicLength thrpt 10 46826.720 ± 4002.887 ops/ms JmhSumLCApplication.fixLength thrpt 10 23.533 ± 3.210 ops/ms {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)