Max Gekk created SPARK-59774:
--------------------------------
Summary: Log at WARN, and name the remedy, when a generated method
is too long to be JIT compiled
Key: SPARK-59774
URL: https://issues.apache.org/jira/browse/SPARK-59774
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
HotSpot does not JIT-compile a method whose bytecode is past 8000 bytes
({{-XX:+DontCompileHugeMethods}}, on by default), so a whole-stage codegen
method past that size runs in the bytecode interpreter for the life of the
executor. Spark detects this: since SPARK-25113 (2.4.0), {{CodeGenerator}} (now
{{CodeCompiler.computeByteCodeStats}}) logs
{noformat}
INFO CodeGenerator: Generated method too long to be JIT compiled:
<class>.<method> is N bytes
{noformat}
But it logs it at INFO. {{spark-shell}} and {{pyspark}} set the root level to
WARN, so an interactive user never sees it, and in a batch job's log it is one
line among thousands of INFO lines. Nothing acts on it by default either:
{{spark.sql.codegen.hugeMethodLimit}}, which falls back from whole-stage
codegen for such a stage, defaults to 65535, and a compiled method cannot
exceed that, so at the default the fallback never fires.
The cost is large. On stock Spark 4.2.0, a projection whose consume method
grows from 7945 to 8677 bytes steps from about 770 to about 4100 ns a row (JDK
17, 21 and 25 alike), and a filter of date ranges over a scan, the
partition-key filter shape of TPC-DS {{modified-q3}}, steps about 135 times,
because the filter is generated into {{processNext}} and so the whole scan loop
runs interpreted. With {{spark.sql.codegen.hugeMethodLimit=8000}} the
projection's step shrinks to about 1.3 times.
Proposal: log the line at WARN, and say what it means and what can be done:
that the method runs interpreted, and that in whole-stage codegen
{{spark.sql.codegen.hugeMethodLimit}} set to the JVM's limit runs such a stage
without whole-stage codegen instead. The line is logged once per compiled class
(compilation is cached), so the extra WARN output is bounded by the number of
distinct generated classes past the limit, which is rare: in the TPC-DS and
TPC-H plans only {{modified-q3}}'s stage crosses it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
