Mihai has explained well why we use the “big hammer” of the executor. As a compiler, you can’t afford to take shortcuts on semantics. It makes sense to use the “real” executor when doing constant reduction, and the price we pay is increased compilation latency.
The “real” executor is a compiler — it produces byte code that needs to be loaded before execution, and execution requires an environment (“container”) that takes a few cycles to initialize — but we could add an interpreter. An interpreter uses the AST as a program, and therefore doesn’t require byte code to be generated and loaded, and doesn’t require an environment. We could create an expression interpreter. It would solve these class-loading problems, but it would be a big effort to create it. In particular, we need a lot of tests to make sure that it is consistent with the compiler. If anyone would like to do this — and is prepared to put in the effort to do it properly — I would be delighted. Another thing to consider, and much less effort. Gonzalo mentioned that there are 80,000 instances of ByteArrayClassLoader. That seems like way too many, even if you are running a benchmark at 1,000 queries per second. (Maybe GC hasn’t gotten to those instances yet, but I doubt it.) I suspect that you can get an easy win by reducing that number. Make sure there is not a leak, and consider using a pool of class-loaders. Maybe create an ExecutorService with a few dozen executors in it, each used in a single thread, reading work from a queue, and clearing itself up after each invocation so that it can be used again. Please log Jira cases for any tasks that you think would make sense. Julian > On Jul 4, 2025, at 11:32 AM, Mihai Budiu <[email protected]> wrote: > > The executor is a big hammer for simplifications. > > The main benefit of using the executor is that there is a single source of > truth: whatever the executor says an expression should evaluate to, this is > the result. > > There are two classes of simplifications: (1) the ones that should be applied > during construction by the builder - these are expected to be simple, and (2) > complex optimizations, that can be more general than just constant folding, > which should be available as separate optimization passes. > > I think it should be feasible to reimplement the simple optimizations without > resorting to the executor. > > Let me point out another problem with the coupling between the executor and > the simplifications: different SQL dialects have different semantics for > various operations. The executor chooses one fixed semantics, and leaves you > no choice. Examples are: division by zero (returns NULL in sqlite), modulo by > a negative number, how operations that require implicit casts work (for casts > that remain implicit and are not inserted by the validator), etc. The > executor is not parameterizable like the validator is, so you cannot > implement alternative choices. > > Mihai > > ________________________________ > From: Stamatis Zampetakis <[email protected]> > Sent: Friday, July 4, 2025 2:20 AM > To: [email protected] <[email protected]> > Subject: Re: Reduce class loading during query optimizing > > Here are a few thoughts that could lead to some useful contributions > to Calcite as well. > > The fact that RexSimplify calls RexExecutor is not ideal. Personally, > I feel that it should be completely independent from one the other. > Most often (e.g., in the ReduceExpressionsRule) before we apply the > expression simplification (RexSimplify) we do constant > reduction/folding (RexExecutor) so in that sense we don't need to > embed the executor inside the simplifier. At the moment, the > RexSimplify class is using RexExecutor only in a very specific case > (simply CAST expressions) so it may be a good time to completely > dissociate the two components before the coupling increases. > > The current implementation of the RexExecutor relies on code > generation/compilation to evaluate expressions (RexNode). The > alternative to that is interpretation but there is no such > implementation currently in Calcite. There are always pros and cons > between compilation and interpretation and the choice always depends > on the use-case. In general, I feel that it would be useful if Calcite > provided an interpreter for row expressions (RexNode). Concretely, I > think this would mean implementing the ScalarCompiler [2] interface > using interpretation instead of code generation. This contribution > would be valuable for RexExecutor but also for the Bindable convention > of Calcite that relies on interpretation. > > The overhead of compilation and class loading has also been observed > in the past and for this reason we have added a cache layer [3] that > can avoid this for frequently appearing queries. I guess the caching > does not take effect for RexExecutor but it could make sense if there > are frequently reappearing expressions. > > All in all, I see at least three contribution areas that could help > with the use-case encountered in Apache Pinot and at the same time > improve Calcite and potentially other projects as well. It would be > nice to see some of them land in the main Calcite repo :) > > Best, > Stamatis > > [1] > https://github.com/apache/calcite/blob/e536d3949674cbbc0fd1162b3fccb211ca75789b/core/src/main/java/org/apache/calcite/rel/rules/ReduceExpressionsRule.java > [2] > http://github.com/apache/calcite/blob/main/core/src/main/java/org/apache/calcite/interpreter/Interpreter.java#L496 > [3] > https://github.com/apache/calcite/blob/e536d3949674cbbc0fd1162b3fccb211ca75789b/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableInterpretable.java#L100 > > On Thu, Jul 3, 2025 at 7:36 PM Mihai Budiu <[email protected]> wrote: >> >> Another thing you could try is to build a RexExecutor which does nothing and >> use that in the RexSimplify class. That may be simpler. >> >> Mihai >> >> ________________________________ >> From: Gonzalo Ortiz Jaureguizar <[email protected]> >> Sent: Thursday, July 3, 2025 4:42 AM >> To: [email protected] <[email protected]> >> Subject: Reduce class loading during query optimizing >> >> Hi there, >> >> Here at Apache Pinot, we utilize Apache Calcite for query optimization. >> Once the query is optimized at logical level (pushing predicates, >> simplifying expressions, etc), we transform the query tree to our own nodes >> and distribute the execution between different nodes. >> >> But we are having some issues in cases where we have 1000 queries per >> second. Specifically, we found that in these cases, we may end up with >>> 80000 instances of >> org.codehaus.commons.compiler.util.reflect.ByteArrayClassLoader instances >> and, even worse, several query threads have to block for the lock of >> jdk.internal.loader.BuiltinClassLoader.loadClassOrNull. >> >> In these two cases, the stacktrace points to Janino. Specifically, Janino >> is being used to compile trivial expressions when queries are simplified. >> Is that necessary or even desirable? We may be doing something wrong (our >> Calcite knowledge is not as good as we would like), but it sounds expensive >> to generate bytecode to run code that will be executed only once. Is there >> a way to disable Janino for these cases? >> >> For the context, >> >> Stack trace of the thread allocating a new classloader: >> ``` >> >> <init>:57, ByteArrayClassLoader (org.codehaus.commons.compiler.util.reflect) >> run:357, SimpleCompiler$2 (org.codehaus.janino) >> run:351, SimpleCompiler$2 (org.codehaus.janino) >> doPrivileged:74, AccessController (java.security) >> getClassLoader2:351, SimpleCompiler (org.codehaus.janino) >> getClassLoader:341, SimpleCompiler (org.codehaus.janino) >> cook:308, ClassBodyEvaluator (org.codehaus.janino) >> cook2:297, ClassBodyEvaluator (org.codehaus.janino) >> cook:273, ClassBodyEvaluator (org.codehaus.janino) >> compile:64, RexExecutable (org.apache.calcite.rex) >> <init>:53, RexExecutable (org.apache.calcite.rex) >> reduce:144, RexExecutorImpl (org.apache.calcite.rex) >> simplifyCast:2304, RexSimplify (org.apache.calcite.rex) >> simplify:293, RexSimplify (org.apache.calcite.rex) >> lambda$simplifyList$2:682, RexSimplify (org.apache.calcite.rex) >> apply:-1, RexSimplify$$Lambda/0x00007faed7bfa218 (org.apache.calcite.rex) >> replaceAllRange:1803, ArrayList (java.util) >> replaceAll:1793, ArrayList (java.util) >> simplifyList:682, RexSimplify (org.apache.calcite.rex) >> simplifyComparison:523, RexSimplify (org.apache.calcite.rex) >> simplifyComparison:515, RexSimplify (org.apache.calcite.rex) >> simplify:311, RexSimplify (org.apache.calcite.rex) >> lambda$simplifyList$2:682, RexSimplify (org.apache.calcite.rex) >> apply:-1, RexSimplify$$Lambda/0x00007faed7bfa218 (org.apache.calcite.rex) >> replaceAllRange:1803, ArrayList (java.util) >> replaceAll:1793, ArrayList (java.util) >> simplifyList:682, RexSimplify (org.apache.calcite.rex) >> simplifyAnd:1529, RexSimplify (org.apache.calcite.rex) >> simplify:282, RexSimplify (org.apache.calcite.rex) >> simplifyUnknownAs:251, RexSimplify (org.apache.calcite.rex) >> simplifyUnknownAsFalse:240, RexSimplify (org.apache.calcite.rex) >> simplifyFilterPredicates:2898, RexSimplify (org.apache.calcite.rex) >> filter:1921, RelBuilder (org.apache.calcite.tools) >> filter:1886, RelBuilder (org.apache.calcite.tools) >> convertProject:393, PushProjector (org.apache.calcite.rel.rules) >> onMatch:179, ProjectFilterTransposeRule (org.apache.calcite.rel.rules) >> fireRule:350, AbstractRelOptPlanner (org.apache.calcite.plan) >> applyRule:541, HepPlanner (org.apache.calcite.plan.hep) >> depthFirstApply:370, HepPlanner (org.apache.calcite.plan.hep) >> depthFirstApply:384, HepPlanner (org.apache.calcite.plan.hep) >> depthFirstApply:384, HepPlanner (org.apache.calcite.plan.hep) >> depthFirstApply:384, HepPlanner (org.apache.calcite.plan.hep) >> applyRules:436, HepPlanner (org.apache.calcite.plan.hep) >> executeRuleCollection:285, HepPlanner (org.apache.calcite.plan.hep) >> execute:105, HepInstruction$RuleCollection$State >> (org.apache.calcite.plan.hep) >> lambda$executeProgram$0:210, HepPlanner (org.apache.calcite.plan.hep) >> accept:-1, HepPlanner$$Lambda/0x00007faed7c20f90 >> (org.apache.calcite.plan.hep) >> forEach:423, ImmutableList (com.google.common.collect) >> executeProgram:209, HepPlanner (org.apache.calcite.plan.hep) >> execute:118, HepProgram$State (org.apache.calcite.plan.hep) >> executeProgram:204, HepPlanner (org.apache.calcite.plan.hep) >> findBestExp:190, HepPlanner (org.apache.calcite.plan.hep) >> optimize:463, QueryEnvironment (org.apache.pinot.query) >> compileQuery:356, QueryEnvironment (org.apache.pinot.query) >> compile:283, QueryEnvironment (org.apache.pinot.query) >> compile:261, QueryEnvironment (org.apache.pinot.query) >> getTableNames:225, PinotQueryResource >> (org.apache.pinot.controller.api.resources) >> ... >> >> ``` >> >> Stack trace of the threads being blocked: >> >> ``` >> ## Thread has lock >> "multi-stage-query-compile-executor-2-thread-1" #374 [369] prio=5 os_prio=0 >> cpu=4267041.58ms elapsed=114126.21s tid=0x00007f7a3d1f5810 nid=369 runnable >> [0x00007f79c517b000] >> java.lang.Thread.State: RUNNABLE >> at java.lang.ClassLoader.findBootstrapClass([email protected]/Native >> Method) >> at java.lang.ClassLoader.findBootstrapClassOrNull([email protected] >> /ClassLoader.java:1277) >> at java.lang.System$2.findBootstrapClassOrNull([email protected] >> /System.java:2397) >> at >> jdk.internal.loader.ClassLoaders$BootClassLoader.loadClassOrNull([email protected] >> /ClassLoaders.java:140) >> at >> jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected] >> /BuiltinClassLoader.java:700) >> at >> jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected] >> /BuiltinClassLoader.java:676) >> - locked <0x000000030a1d87f8> (a java.lang.Object) >> at >> jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected] >> /BuiltinClassLoader.java:700) >> at >> jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected] >> /BuiltinClassLoader.java:676) >> - locked <0x000000030ab04140> (a java.lang.Object) >> at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected] >> /BuiltinClassLoader.java:639) >> at >> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected] >> /ClassLoaders.java:188) >> at java.lang.ClassLoader.loadClass([email protected] >> /ClassLoader.java:526) >> at >> org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:75) >> at >> org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:317) >> - locked <0x00000007f780cc68> (a >> org.codehaus.janino.ClassLoaderIClassLoader) >> at >> org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:9074) >> at >> org.codehaus.janino.UnitCompiler.getRawReferenceType(UnitCompiler.java:7242) >> at >> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:7142) >> at >> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:7023) >> at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6994) >> at >> org.codehaus.janino.UnitCompiler.access$14900(UnitCompiler.java:240) >> at >> org.codehaus.janino.UnitCompiler$24.visitReferenceType(UnitCompiler.java:6891) >> at >> org.codehaus.janino.UnitCompiler$24.visitReferenceType(UnitCompiler.java:6888) >> >> ...... >> >> at >> org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:281) >> at >> org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:250) >> at >> org.apache.calcite.rex.RexSimplify.simplifyUnknownAsFalse(RexSimplify.java:239) >> at >> org.apache.calcite.rex.RexSimplify.simplifyFilterPredicates(RexSimplify.java:2810) >> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1809) >> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1774) >> at >> org.apache.calcite.rel.rules.PushProjector.convertProject(PushProjector.java:394) >> at >> org.apache.calcite.rel.rules.ProjectFilterTransposeRule.onMatch(ProjectFilterTransposeRule.java:179) >> at >> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337) >> at >> org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556) >> at >> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:371) >> at >> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:385) >> at >> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:385) >> at >> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:385) >> at >> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:437) >> at >> org.apache.calcite.plan.hep.HepPlanner.executeRuleCollection(HepPlanner.java:286) >> at >> org.apache.calcite.plan.hep.HepInstruction$RuleCollection$State.execute(HepInstruction.java:105) >> at >> org.apache.calcite.plan.hep.HepPlanner.lambda$executeProgram$0(HepPlanner.java:211) >> at >> org.apache.calcite.plan.hep.HepPlanner$$Lambda/0x00007f79fbf0a360.accept(Unknown >> Source) >> at >> org.apache.pinot.shaded.com.google.common.collect.ImmutableList.forEach(ImmutableList.java:423) >> at >> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:210) >> at >> org.apache.calcite.plan.hep.HepProgram$State.execute(HepProgram.java:118) >> at >> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:205) >> at >> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:191) >> at >> org.apache.pinot.query.QueryEnvironment.optimize(QueryEnvironment.java:394) >> at >> org.apache.pinot.query.QueryEnvironment.compileQuery(QueryEnvironment.java:347) >> at >> org.apache.pinot.query.QueryEnvironment.planQuery(QueryEnvironment.java:187) >> at >> org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.lambda$handleRequest$2(MultiStageBrokerRequestHandler.java:222) >> >> >> >> >> >> Waiting threads: >> "multi-stage-query-compile-executor-2-thread-3" #376 [371] prio=5 os_prio=0 >> cpu=4261107.12ms elapsed=114126.12s tid=0x00007f79c7d8b010 nid=371 waiting >> for monitor entry [0x00007f79c48fb000] >> java.lang.Thread.State: BLOCKED (on object monitor) >> at >> jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected] >> /BuiltinClassLoader.java:651) >> - waiting to lock <0x000000030ab04140> (a java.lang.Object) >> at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected] >> /BuiltinClassLoader.java:639) >> at >> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected] >> /ClassLoaders.java:188) >> at java.lang.ClassLoader.loadClass([email protected] >> /ClassLoader.java:526) >> at >> org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:75) >> at >> org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:317) >> - locked <0x00000007f70a2398> (a >> org.codehaus.janino.ClassLoaderIClassLoader) >> at >> org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:9074) >> at >> org.codehaus.janino.UnitCompiler.getRawReferenceType(UnitCompiler.java:7242) >> at >> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:7142) >> at >> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:7023) >> at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6994) >> at >> org.codehaus.janino.UnitCompiler.access$14900(UnitCompiler.java:240) >> at >> org.codehaus.janino.UnitCompiler$24.visitReferenceType(UnitCompiler.java:6891) >> at >> org.codehaus.janino.UnitCompiler$24.visitReferenceType(UnitCompiler.java:6888) >> at org.codehaus.janino.Java$ReferenceType.accept(Java.java:4289) >> at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6888) >> at >> org.codehaus.janino.UnitCompiler.getRawType(UnitCompiler.java:6884) >> at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:7366) >> >> "multi-stage-query-compile-executor-2-thread-4" #396 [391] prio=5 os_prio=0 >> cpu=4264527.65ms elapsed=114125.23s tid=0x00007f79cdbc2010 nid=391 waiting >> for monitor entry [0x00007f79c12fb000] >> java.lang.Thread.State: BLOCKED (on object monitor) >> at >> jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected] >> /BuiltinClassLoader.java:651) >> - waiting to lock <0x000000030ab04140> (a java.lang.Object) >> at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected] >> /BuiltinClassLoader.java:639) >> at >> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected] >> /ClassLoaders.java:188) >> at java.lang.ClassLoader.loadClass([email protected] >> /ClassLoader.java:526) >> at >> org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:75) >> at >> org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:317) >> - locked <0x00000007f880ca18> (a >> org.codehaus.janino.ClassLoaderIClassLoader) >> at >> org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:9074) >> at >> org.codehaus.janino.UnitCompiler.getRawReferenceType(UnitCompiler.java:7242) >> at >> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:7142) >> at >> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:7023) >> at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6994) >> at >> org.codehaus.janino.UnitCompiler.access$14900(UnitCompiler.java:240) >> at >> org.codehaus.janino.UnitCompiler$24.visitReferenceType(UnitCompiler.java:6891) >> at >> org.codehaus.janino.UnitCompiler$24.visitReferenceType(UnitCompiler.java:6888) >> at org.codehaus.janino.Java$ReferenceType.accept(Java.java:4289) >> at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6888) >> at >> org.codehaus.janino.UnitCompiler.getRawType(UnitCompiler.java:6884) >> at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:7366) >> ```
