Yaoxuan Wu created SPARK-60362:
----------------------------------
Summary: RC pushdown of IS NULL on a nested field drops rows whose
parent struct is NULL
Key: SPARK-60362
URL: https://issues.apache.org/jira/browse/SPARK-60362
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0, 4.0.0
Reporter: Yaoxuan Wu
{code:python}
path = "/tmp/orc_nested_is_null"
spark.sql("SELECT IF(id = 0, NULL, named_struct('b', id)) AS s FROM range(2)") \
.coalesce(1).write.mode("overwrite").orc(path)
spark.read.orc(path).filter("s.b IS NULL").collect()
# expected: [Row(s=None)]
# actual: []
spark.conf.set("spark.sql.orc.filterPushdown", "false")
spark.read.orc(path).filter("s.b IS NULL").collect()
# [Row(s=None)] correct without ORC filter pushdown
{code}
When {{s}} is NULL, {{s.b}} is NULL as well, so the row should be returned. It
is returned once ORC filter pushdown is turned off, and with Parquet.
{{s.b IS NULL}} is pushed to ORC, which evaluates it on the statistics of the
child column {{s.b}}. A row whose parent {{s}} is NULL has no entry in the
child column, so the child's statistics show no NULLs and the row group is
skipped.
Reproduces on 3.5.8, 4.0.0, 4.1.1 and 4.2.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
