dwsmith1983 commented on code in PR #5654:
URL: https://github.com/apache/datafusion-comet/pull/5654#discussion_r4100272498
##########
spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala:
##########
@@ -2079,10 +2079,122 @@ abstract class ParquetReadSuite extends CometTestBase {
}
}
- // Verbatim port of Spark `ParquetFieldIdIOSuite.test("multiple id
matches")` so the shim
- // error path is exercised on both 3.x and 4.x. The stock suite is the CI
signal but it
- // requires the Spark test jars and `withAllParquetReaders`; keeping a copy
here lets us
- // iterate locally.
+ // The shape a schema evolution leaves behind: a nested column dropped and
added back under
+ // its old name gets a fresh field id, so the file holds `struct<x (id 1), y
(id 2)>` while
+ // the table reads `struct<x (id 3), y (id 2)>`. The names line up at every
position, which
+ // is exactly what the metadata-only relabel shortcut in the native cast
looks for, so without
+ // the field-mapping gate the scan hands back the old x values. Spark
returns null for x and
+ // keeps y. One test per nesting, so a wrong answer names the level that
produced it.
+ // Before Spark 4.1 the vectorized reader raises on this read below a list
or map, since its
+ // column vector rejects the placeholder field the clipped schema carries
for the unmatched
+ // id, so the comparison with Spark runs from 4.1 on. The pinned rows hold
everywhere.
+ private def checkDroppedAndReAddedFieldId(
Review Comment:
Done. One test reads `s`, `l` and `m` under two read schemas, `x (id 3), y
(id 2)` for the dropped-and-re-added shape and `x (id 2), y (id 1)` for the
swapped one, so the Scala side now covers both shapes in the issue. The
expected rows are derived from the written rows through the per-case remap,
null for the re-added `x` and the two values exchanged for the swap, and pinned
with `checkAnswer` after the comparison with Spark. The comparison with Spark
below a list or map starts at 4.1 for both shapes: on 3.5 Spark's own
vectorized reader rejects the clipped struct in `ParquetColumnVector` for the
swapped ids too, since the clip carries the file's field order there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
