[
https://issues.apache.org/jira/browse/SPARK-60048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60048:
-----------------------------------
Labels: pull-request-available (was: )
> Remember a whole-stage codegen split decision by a key that ignores literal
> values
> ----------------------------------------------------------------------------------
>
> Key: SPARK-60048
> URL: https://issues.apache.org/jira/browse/SPARK-60048
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Max Gekk
> Priority: Major
> Labels: pull-request-available
>
> Whole-stage codegen can split the expressions of a stage that has a method
> past the JIT limit (SPARK-33301, https://github.com/apache/spark/pull/59225).
> It decides by generating the stage's code in one piece, compiling it as a
> trial, and generating it again with some expressions split. The decision is
> remembered by WholeStageCodegenExec.decisionKey, a digest of the generated
> code with the class loader, the compiler, the method limit and
> methodSplitThreshold, so that executing a query again skips the trial
> compiles.
> A literal that changes on every run changes that digest on every run.
> ComputeCurrentTime turns current_timestamp() into a literal and
> Literal.doGenCode writes it into the source, so a stage holding one decides
> afresh each run and pays two code generations and two compiles where master
> pays one of each. Queries without such literals compile nothing new after the
> first execution.
> Proposed: remember the decision by a key that ignores literal values, either
> the code digested with numeric literals masked, or the plan with literals
> normalized plus every conf that affects code generation.
> Care is needed: a literal's value moves a method's bytecode size by a byte or
> two, which can matter for a method near the limit, and a key built from the
> plan has to carry all the codegen confs. Tests: two executions of a query
> with current_timestamp() decide once; a decision is not reused under a
> different conf that changes the code.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
