jaylisde commented on code in PR #12067:
URL: https://github.com/apache/gluten/pull/12067#discussion_r3353274886
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenMapStatusEndToEndSuite.scala:
##########
@@ -28,5 +29,58 @@ class GlutenMapStatusEndToEndSuite extends
MapStatusEndToEndSuite with GlutenTes
_spark.sparkContext.conf
.set(SQLConf.CLASSIC_SHUFFLE_DEPENDENCY_FILE_CLEANUP_ENABLED.key,
"false")
_spark.conf.set(SQLConf.CLASSIC_SHUFFLE_DEPENDENCY_FILE_CLEANUP_ENABLED.key,
"false")
+
+ // Disable ANSI fallback to force Gluten's ColumnarShuffleWriter path.
+ _spark.conf.set("spark.gluten.sql.ansiFallback.enabled", "false")
+ }
+
+ import org.apache.spark.MapOutputTrackerMaster
+
+ private def getLatestShuffleChecksumValues(): Array[Long] = {
+ val tracker = _spark.sparkContext.env.mapOutputTracker
+ .asInstanceOf[MapOutputTrackerMaster]
+ val latestShuffleId = tracker.shuffleStatuses.keys.max
+ tracker.shuffleStatuses(latestShuffleId).mapStatuses.map(_.checksumValue)
+ }
+
+ test("Gluten row-based checksum is deterministic") {
Review Comment:
Done - moved the two Gluten-specific tests into a new
GlutenRowBasedChecksumSuite under `org.apache.spark.sql.gluten`.
GlutenMapStatusEndToEndSuite is restored to a thin wrapper that only runs
upstream Spark's MapStatusEndToEndSuite tests under Gluten. The new suite
extends GlutenSQLTestsTrait directly and configures the required `SQLConfs /
GLUTEN_ANSI_FALLBACK_ENABLED` via sparkConf. Thanks for the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
