Hi there,
Here at Apache Pinot, we utilize Apache Calcite for query optimization.
Once the query is optimized at logical level (pushing predicates,
simplifying expressions, etc), we transform the query tree to our own nodes
and distribute the execution between different nodes.
But we are having some issues in cases where we have 1000 queries per
second. Specifically, we found that in these cases, we may end up with
>80000 instances of
org.codehaus.commons.compiler.util.reflect.ByteArrayClassLoader instances
and, even worse, several query threads have to block for the lock of
jdk.internal.loader.BuiltinClassLoader.loadClassOrNull.
In these two cases, the stacktrace points to Janino. Specifically, Janino
is being used to compile trivial expressions when queries are simplified.
Is that necessary or even desirable? We may be doing something wrong (our
Calcite knowledge is not as good as we would like), but it sounds expensive
to generate bytecode to run code that will be executed only once. Is there
a way to disable Janino for these cases?
For the context,
Stack trace of the thread allocating a new classloader:
```
<init>:57, ByteArrayClassLoader (org.codehaus.commons.compiler.util.reflect)
run:357, SimpleCompiler$2 (org.codehaus.janino)
run:351, SimpleCompiler$2 (org.codehaus.janino)
doPrivileged:74, AccessController (java.security)
getClassLoader2:351, SimpleCompiler (org.codehaus.janino)
getClassLoader:341, SimpleCompiler (org.codehaus.janino)
cook:308, ClassBodyEvaluator (org.codehaus.janino)
cook2:297, ClassBodyEvaluator (org.codehaus.janino)
cook:273, ClassBodyEvaluator (org.codehaus.janino)
compile:64, RexExecutable (org.apache.calcite.rex)
<init>:53, RexExecutable (org.apache.calcite.rex)
reduce:144, RexExecutorImpl (org.apache.calcite.rex)
simplifyCast:2304, RexSimplify (org.apache.calcite.rex)
simplify:293, RexSimplify (org.apache.calcite.rex)
lambda$simplifyList$2:682, RexSimplify (org.apache.calcite.rex)
apply:-1, RexSimplify$$Lambda/0x00007faed7bfa218 (org.apache.calcite.rex)
replaceAllRange:1803, ArrayList (java.util)
replaceAll:1793, ArrayList (java.util)
simplifyList:682, RexSimplify (org.apache.calcite.rex)
simplifyComparison:523, RexSimplify (org.apache.calcite.rex)
simplifyComparison:515, RexSimplify (org.apache.calcite.rex)
simplify:311, RexSimplify (org.apache.calcite.rex)
lambda$simplifyList$2:682, RexSimplify (org.apache.calcite.rex)
apply:-1, RexSimplify$$Lambda/0x00007faed7bfa218 (org.apache.calcite.rex)
replaceAllRange:1803, ArrayList (java.util)
replaceAll:1793, ArrayList (java.util)
simplifyList:682, RexSimplify (org.apache.calcite.rex)
simplifyAnd:1529, RexSimplify (org.apache.calcite.rex)
simplify:282, RexSimplify (org.apache.calcite.rex)
simplifyUnknownAs:251, RexSimplify (org.apache.calcite.rex)
simplifyUnknownAsFalse:240, RexSimplify (org.apache.calcite.rex)
simplifyFilterPredicates:2898, RexSimplify (org.apache.calcite.rex)
filter:1921, RelBuilder (org.apache.calcite.tools)
filter:1886, RelBuilder (org.apache.calcite.tools)
convertProject:393, PushProjector (org.apache.calcite.rel.rules)
onMatch:179, ProjectFilterTransposeRule (org.apache.calcite.rel.rules)
fireRule:350, AbstractRelOptPlanner (org.apache.calcite.plan)
applyRule:541, HepPlanner (org.apache.calcite.plan.hep)
depthFirstApply:370, HepPlanner (org.apache.calcite.plan.hep)
depthFirstApply:384, HepPlanner (org.apache.calcite.plan.hep)
depthFirstApply:384, HepPlanner (org.apache.calcite.plan.hep)
depthFirstApply:384, HepPlanner (org.apache.calcite.plan.hep)
applyRules:436, HepPlanner (org.apache.calcite.plan.hep)
executeRuleCollection:285, HepPlanner (org.apache.calcite.plan.hep)
execute:105, HepInstruction$RuleCollection$State (org.apache.calcite.plan.hep)
lambda$executeProgram$0:210, HepPlanner (org.apache.calcite.plan.hep)
accept:-1, HepPlanner$$Lambda/0x00007faed7c20f90 (org.apache.calcite.plan.hep)
forEach:423, ImmutableList (com.google.common.collect)
executeProgram:209, HepPlanner (org.apache.calcite.plan.hep)
execute:118, HepProgram$State (org.apache.calcite.plan.hep)
executeProgram:204, HepPlanner (org.apache.calcite.plan.hep)
findBestExp:190, HepPlanner (org.apache.calcite.plan.hep)
optimize:463, QueryEnvironment (org.apache.pinot.query)
compileQuery:356, QueryEnvironment (org.apache.pinot.query)
compile:283, QueryEnvironment (org.apache.pinot.query)
compile:261, QueryEnvironment (org.apache.pinot.query)
getTableNames:225, PinotQueryResource
(org.apache.pinot.controller.api.resources)
...
```
Stack trace of the threads being blocked:
```
## Thread has lock
"multi-stage-query-compile-executor-2-thread-1" #374 [369] prio=5 os_prio=0
cpu=4267041.58ms elapsed=114126.21s tid=0x00007f7a3d1f5810 nid=369 runnable
[0x00007f79c517b000]
java.lang.Thread.State: RUNNABLE
at java.lang.ClassLoader.findBootstrapClass([email protected]/Native
Method)
at java.lang.ClassLoader.findBootstrapClassOrNull([email protected]
/ClassLoader.java:1277)
at java.lang.System$2.findBootstrapClassOrNull([email protected]
/System.java:2397)
at
jdk.internal.loader.ClassLoaders$BootClassLoader.loadClassOrNull([email protected]
/ClassLoaders.java:140)
at
jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]
/BuiltinClassLoader.java:700)
at
jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]
/BuiltinClassLoader.java:676)
- locked <0x000000030a1d87f8> (a java.lang.Object)
at
jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]
/BuiltinClassLoader.java:700)
at
jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]
/BuiltinClassLoader.java:676)
- locked <0x000000030ab04140> (a java.lang.Object)
at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected]
/BuiltinClassLoader.java:639)
at
jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected]
/ClassLoaders.java:188)
at java.lang.ClassLoader.loadClass([email protected]
/ClassLoader.java:526)
at
org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:75)
at
org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:317)
- locked <0x00000007f780cc68> (a
org.codehaus.janino.ClassLoaderIClassLoader)
at
org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:9074)
at
org.codehaus.janino.UnitCompiler.getRawReferenceType(UnitCompiler.java:7242)
at
org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:7142)
at
org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:7023)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6994)
at
org.codehaus.janino.UnitCompiler.access$14900(UnitCompiler.java:240)
at
org.codehaus.janino.UnitCompiler$24.visitReferenceType(UnitCompiler.java:6891)
at
org.codehaus.janino.UnitCompiler$24.visitReferenceType(UnitCompiler.java:6888)
......
at
org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:281)
at
org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:250)
at
org.apache.calcite.rex.RexSimplify.simplifyUnknownAsFalse(RexSimplify.java:239)
at
org.apache.calcite.rex.RexSimplify.simplifyFilterPredicates(RexSimplify.java:2810)
at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1809)
at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1774)
at
org.apache.calcite.rel.rules.PushProjector.convertProject(PushProjector.java:394)
at
org.apache.calcite.rel.rules.ProjectFilterTransposeRule.onMatch(ProjectFilterTransposeRule.java:179)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337)
at
org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:371)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:385)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:385)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:385)
at
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:437)
at
org.apache.calcite.plan.hep.HepPlanner.executeRuleCollection(HepPlanner.java:286)
at
org.apache.calcite.plan.hep.HepInstruction$RuleCollection$State.execute(HepInstruction.java:105)
at
org.apache.calcite.plan.hep.HepPlanner.lambda$executeProgram$0(HepPlanner.java:211)
at
org.apache.calcite.plan.hep.HepPlanner$$Lambda/0x00007f79fbf0a360.accept(Unknown
Source)
at
org.apache.pinot.shaded.com.google.common.collect.ImmutableList.forEach(ImmutableList.java:423)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:210)
at
org.apache.calcite.plan.hep.HepProgram$State.execute(HepProgram.java:118)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:205)
at
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:191)
at
org.apache.pinot.query.QueryEnvironment.optimize(QueryEnvironment.java:394)
at
org.apache.pinot.query.QueryEnvironment.compileQuery(QueryEnvironment.java:347)
at
org.apache.pinot.query.QueryEnvironment.planQuery(QueryEnvironment.java:187)
at
org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.lambda$handleRequest$2(MultiStageBrokerRequestHandler.java:222)
Waiting threads:
"multi-stage-query-compile-executor-2-thread-3" #376 [371] prio=5 os_prio=0
cpu=4261107.12ms elapsed=114126.12s tid=0x00007f79c7d8b010 nid=371 waiting
for monitor entry [0x00007f79c48fb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]
/BuiltinClassLoader.java:651)
- waiting to lock <0x000000030ab04140> (a java.lang.Object)
at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected]
/BuiltinClassLoader.java:639)
at
jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected]
/ClassLoaders.java:188)
at java.lang.ClassLoader.loadClass([email protected]
/ClassLoader.java:526)
at
org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:75)
at
org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:317)
- locked <0x00000007f70a2398> (a
org.codehaus.janino.ClassLoaderIClassLoader)
at
org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:9074)
at
org.codehaus.janino.UnitCompiler.getRawReferenceType(UnitCompiler.java:7242)
at
org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:7142)
at
org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:7023)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6994)
at
org.codehaus.janino.UnitCompiler.access$14900(UnitCompiler.java:240)
at
org.codehaus.janino.UnitCompiler$24.visitReferenceType(UnitCompiler.java:6891)
at
org.codehaus.janino.UnitCompiler$24.visitReferenceType(UnitCompiler.java:6888)
at org.codehaus.janino.Java$ReferenceType.accept(Java.java:4289)
at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6888)
at
org.codehaus.janino.UnitCompiler.getRawType(UnitCompiler.java:6884)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:7366)
"multi-stage-query-compile-executor-2-thread-4" #396 [391] prio=5 os_prio=0
cpu=4264527.65ms elapsed=114125.23s tid=0x00007f79cdbc2010 nid=391 waiting
for monitor entry [0x00007f79c12fb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]
/BuiltinClassLoader.java:651)
- waiting to lock <0x000000030ab04140> (a java.lang.Object)
at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected]
/BuiltinClassLoader.java:639)
at
jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected]
/ClassLoaders.java:188)
at java.lang.ClassLoader.loadClass([email protected]
/ClassLoader.java:526)
at
org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:75)
at
org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:317)
- locked <0x00000007f880ca18> (a
org.codehaus.janino.ClassLoaderIClassLoader)
at
org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:9074)
at
org.codehaus.janino.UnitCompiler.getRawReferenceType(UnitCompiler.java:7242)
at
org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:7142)
at
org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:7023)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6994)
at
org.codehaus.janino.UnitCompiler.access$14900(UnitCompiler.java:240)
at
org.codehaus.janino.UnitCompiler$24.visitReferenceType(UnitCompiler.java:6891)
at
org.codehaus.janino.UnitCompiler$24.visitReferenceType(UnitCompiler.java:6888)
at org.codehaus.janino.Java$ReferenceType.accept(Java.java:4289)
at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6888)
at
org.codehaus.janino.UnitCompiler.getRawType(UnitCompiler.java:6884)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:7366)
```
