Max Gekk created SPARK-60872:
--------------------------------
Summary: Test only the null bit of a columnar scan column in an IS
NOT NULL check generated ahead of its first use
Key: SPARK-60872
URL: https://issues.apache.org/jira/browse/SPARK-60872
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
With whole-stage codegen, an IS NOT NULL check on a column of a columnar scan
is generated through evaluateRequiredVariables, which emits the column's whole
input variable: the null bit and the value together (isNull = col.isNullAt(i);
value = isNull ? default : col.getX(i)). So when the check is placed before a
conjunct that can raise an error (SPARK-60329, and SPARK-60315 after a conjunct
that can be NULL), the value is decoded on every row that reaches the check,
although only the null bit is needed there and the value is needed only by a
later conjunct that references the column, on the rows that reach it.
For a wide column, such as a DECIMAL(38, 10) or a BINARY value, the decode is
the cost of the placement: a filter over 20M Parquet rows with a written check
on such a column before a conjunct that can raise took 520 ms instead of 314 ms
when the check was generated at its position. Spark pays it only where the
check moves, before a conjunct that can raise an error, which is why
SPARK-60329 does not move inferred checks and does not move any check before a
conjunct that is known not to raise.
Testing only the null bit at the check's position, and decoding the value at
its first use by a later conjunct, would return the same results without the
extra decodes. It needs a change to how the input variables of a columnar scan
are generated: separate code for the null bit and the value, or a null-only
accessor that the check can use, with the value's code emitted at its first
use. The same would apply to the synthetic IsNotNull(r) checks that FilterExec
generates for a column covered only by a check on an expression.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
