felipepessoto commented on code in PR #13036:
URL: https://github.com/apache/gluten/pull/13036#discussion_r4023987854
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -172,6 +172,33 @@ cherry_pick_delta_fix
46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanRepor
cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more
test suites)"
echo "::endgroup::"
+echo "::group::Capping DeltaParquetFileFormat fixture row groups by row count"
+# DeltaParquetFileFormatSuite generates one 20,000-row Parquet file and sets a
+# 50 KiB block size to ensure that it contains multiple row groups. Velox sizes
+# row groups by compressed buffered bytes, so the highly compressible integer
+# fixture remains a single row group and the DV read tests fail before reaching
+# their assertions. Gluten's native writer also supports parquet.block.rows;
+# cap the fixture at 10,000 rows to create at least two row groups regardless
of
+# compression while keeping the native write path enabled.
+DPFFS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala"
+if [ ! -f "$DPFFS" ]; then
+ echo "Expected file not found in Delta clone: $DPFFS" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
>&2
+ exit 1
+fi
+sed -i '/hadoopConf().set("parquet.block.size", (1024 \* 50).toString)/a\
+ hadoopConf().set("parquet.block.rows", "10000")' "$DPFFS"
+ROW_CAPS=$(grep -c 'hadoopConf().set("parquet.block.rows", "10000")' "$DPFFS"
|| true)
Review Comment:
Fixed in b0b6183b8. The guard now counts the exact adjacent, fully shaped
`parquet.block.size` and `parquet.block.rows` lines instead of counting row-cap
text anywhere in the file. I validated it against Delta v4.2.0 and with a
simulated source-drift case where the expected block-size line moved but an
unrelated row-cap line remained; the latter is correctly rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
