sunchao commented on code in PR #5732:
URL: https://github.com/apache/datafusion-comet/pull/5732#discussion_r3997867923
##########
spark/src/test/scala/org/apache/comet/CometFuzzIcebergSuite.scala:
##########
@@ -237,6 +238,95 @@ class CometFuzzIcebergSuite extends CometFuzzIcebergBase {
}
}
+ test(
+ "filter pushdown - IS NULL/IS NOT NULL on list and map columns stays
native, struct falls back") {
+ val tableName = "hadoop_catalog.db.null_check_test"
+ try {
+ spark.sql(s"""
+ CREATE TABLE $tableName (
+ id INT, l ARRAY<STRUCT<a: INT>>, m MAP<STRING, STRUCT<a: INT>>, s
STRUCT<a: INT>
+ ) USING iceberg
+ """)
+ // Container nullness is distinct from emptiness, null elements and null
struct fields.
+ spark.sql(s"""
+ INSERT INTO $tableName VALUES
+ (1, array(named_struct('a', 1)), map('k', named_struct('a', 1)),
named_struct('a', 1)),
+ (2, NULL, NULL, NULL),
+ (3, array(), map(), named_struct('a', NULL)),
+ (4, array(NULL), map('k', NULL), named_struct('a', NULL)),
+ (5, array(named_struct('a', NULL)), map('k', named_struct('a',
NULL)), named_struct('a', NULL))
+ """)
+ for (column <- Seq("l", "m", "s"); predicate <- Seq("IS NULL", "IS NOT
NULL")) {
+ val query = s"SELECT id FROM $tableName WHERE $column $predicate"
+ withClue(query) {
+ val (_, cometPlan) = checkSparkAnswer(query)
+ val expected = if (predicate == "IS NULL") Seq(Row(2)) else Seq(1,
3, 4, 5).map(Row(_))
+ checkAnswer(spark.sql(query), expected)
+ val expectedScans = if (column == "s") 0 else 1
+ assert(collectIcebergNativeScans(cometPlan).length == expectedScans,
s"$cometPlan")
Review Comment:
### Correctness
[P2] Update the fuzz test's struct scan expectation
Could you update this expectation along with removing the struct-null
fallback? `CometFuzzIcebergSuite` is unchanged from the previous revision, so
both `WHERE s IS NULL` and `WHERE s IS NOT NULL` still require zero native
scans. Its base enables native Iceberg and Comet execution, and this
unpartitioned fixture now qualifies for a native scan. The assertion therefore
rejects the behavior this update enables, even when the returned rows are
correct. The corresponding `CometIcebergNativeSuite` assertions were updated,
but this suite also runs in the Linux and macOS scan groups. Please require one
native scan for all three column types, update the test name, and run this fuzz
regression while retaining the Spark-result and expected-row checks. This is a
source-verified assertion conflict. Current CI has no executed test jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
