[
https://issues.apache.org/jira/browse/SPARK-59774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-59774.
------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59020
[https://github.com/apache/spark/pull/59020]
> Warn once, and name the remedy, when a whole-stage codegen method is too long
> to be JIT compiled
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-59774
> URL: https://issues.apache.org/jira/browse/SPARK-59774
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> HotSpot does not JIT-compile a method whose bytecode is past 8000 bytes
> ({{-XX:+DontCompileHugeMethods}}, on by default), so a whole-stage codegen
> method past that size runs in the bytecode interpreter for the life of the
> executor. Spark detects this: since SPARK-25113 (2.4.0), {{CodeGenerator}}
> (now {{CodeCompiler.computeByteCodeStats}}) logs
> {noformat}
> INFO CodeGenerator: Generated method too long to be JIT compiled:
> <class>.<method> is N bytes
> {noformat}
> But it logs it at INFO. {{spark-shell}} and {{pyspark}} set the root level to
> WARN, so an interactive user never sees it, and in a batch job's log it is
> one line among thousands of INFO lines. Nothing acts on it by default either:
> {{spark.sql.codegen.hugeMethodLimit}}, which falls back from whole-stage
> codegen for such a stage, defaults to 65535, and a compiled method cannot
> exceed that, so at the default the fallback never fires.
> The cost is large. On stock Spark 4.2.0, a projection whose consume method
> grows from 7945 to 8677 bytes steps from about 770 to about 4100 ns a row
> (JDK 17, 21 and 25 alike), and a filter of date ranges over a scan, the
> partition-key filter shape of TPC-DS {{modified-q3}}, steps about 135 times,
> because the filter is generated into {{processNext}} and so the whole scan
> loop runs interpreted. With {{spark.sql.codegen.hugeMethodLimit=8000}} the
> projection's step shrinks to about 1.3 times.
> Proposal: log the line at WARN, and say what it means and what can be done:
> that the method runs interpreted, and that in whole-stage codegen
> {{spark.sql.codegen.hugeMethodLimit}} set to the JVM's limit runs such a
> stage without whole-stage codegen instead. The line is logged once per
> compiled class (compilation is cached), so the extra WARN output is bounded
> by the number of distinct generated classes past the limit, which is rare: in
> the TPC-DS and TPC-H plans only {{modified-q3}}'s stage crosses it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
