Yu Xu created CALCITE-7666:
------------------------------
Summary: Make regex pattern caches static to avoid repeated
allocation in RexInterpreter
Key: CALCITE-7666
URL: https://issues.apache.org/jira/browse/CALCITE-7666
Project: Calcite
Issue Type: Improvement
Components: core
Affects Versions: 1.42.0
Reporter: Yu Xu
Assignee: Yu Xu
Fix For: 1.43.0
Recently, during a performance test of Simpifly, I discovered a memory
performance issue:
{code:java}
SELECT * FROM t
WHERE v0 AND (v0 OR v1 OR v2 OR ... OR v9) -- 10 boolean expression
AND (i0 = 0 OR i0 = 1 OR i0 = 2 OR i0 = 3) -- 10 integer expression
AND (s0 = 'foo' OR s0 = 'bar') -- 1 string expression
{code}
Tests of this kind generally cannot complete successfully because the data
volume is massive: 2^10 (Boolean) × 4^10 (Integer) × 2^1 (String) = 21,233,664
combinations
Based on this test, I analyzed the memory status of the process:
It is primarily caused by the following caches:
SqlFunctions.LikeFunction
SqlFunctions.SimilarFunction
SqlFunctions.SimilarEscapeFunction
{code:java}
private final LoadingCache<Key, Pattern> cache =
CacheBuilder.newBuilder()
.maximumSize(FUNCTION_LEVEL_CACHE_MAX_SIZE.value())
.build(CacheLoader.from(Key::toPattern)); {code}
When tests frequently call `new RexInterpreter()`, a new `LikeFunction()` or
`SimilarFunction()` is instantiated each time, and each of these constructors,
in turn, creates a separate Guava `LoadingCache`.
When optimized the cache to be static, memory overhead was significantly
reduced.The comparison results are as follows:
||metrics ||current||after change||ratio||
|LocalCache instance|134,586|0|↓100%|
|Memory usage|677M |285M|↓58%|
|Memory compression|259M|36M|↓86%|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
