Max Gekk created SPARK-60048:
--------------------------------
Summary: Remember a whole-stage codegen split decision by a key
that ignores literal values
Key: SPARK-60048
URL: https://issues.apache.org/jira/browse/SPARK-60048
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
Whole-stage codegen can split the expressions of a stage that has a method past
the JIT limit (SPARK-33301, https://github.com/apache/spark/pull/59225). It
decides by generating the stage's code in one piece, compiling it as a trial,
and generating it again with some expressions split. The decision is remembered
by WholeStageCodegenExec.decisionKey, a digest of the generated code with the
class loader, the compiler, the method limit and methodSplitThreshold, so that
executing a query again skips the trial compiles.
A literal that changes on every run changes that digest on every run.
ComputeCurrentTime turns current_timestamp() into a literal and
Literal.doGenCode writes it into the source, so a stage holding one decides
afresh each run and pays two code generations and two compiles where master
pays one of each. Queries without such literals compile nothing new after the
first execution.
Proposed: remember the decision by a key that ignores literal values, either
the code digested with numeric literals masked, or the plan with literals
normalized plus every conf that affects code generation.
Care is needed: a literal's value moves a method's bytecode size by a byte or
two, which can matter for a method near the limit, and a key built from the
plan has to carry all the codegen confs. Tests: two executions of a query with
current_timestamp() decide once; a decision is not reused under a different
conf that changes the code.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
