philo-he commented on code in PR #12067:
URL: https://github.com/apache/gluten/pull/12067#discussion_r3346625358
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenMapStatusEndToEndSuite.scala:
##########
@@ -28,5 +29,58 @@ class GlutenMapStatusEndToEndSuite extends
MapStatusEndToEndSuite with GlutenTes
_spark.sparkContext.conf
.set(SQLConf.CLASSIC_SHUFFLE_DEPENDENCY_FILE_CLEANUP_ENABLED.key,
"false")
_spark.conf.set(SQLConf.CLASSIC_SHUFFLE_DEPENDENCY_FILE_CLEANUP_ENABLED.key,
"false")
+
+ // Disable ANSI fallback to force Gluten's ColumnarShuffleWriter path.
+ _spark.conf.set("spark.gluten.sql.ansiFallback.enabled", "false")
+ }
+
+ import org.apache.spark.MapOutputTrackerMaster
+
+ private def getLatestShuffleChecksumValues(): Array[Long] = {
+ val tracker = _spark.sparkContext.env.mapOutputTracker
+ .asInstanceOf[MapOutputTrackerMaster]
+ val latestShuffleId = tracker.shuffleStatuses.keys.max
+ tracker.shuffleStatuses(latestShuffleId).mapStatuses.map(_.checksumValue)
+ }
+
+ test("Gluten row-based checksum is deterministic") {
Review Comment:
Test suites like GlutenMapStatusEndToEndSuite extend Spark's own test
classes to run Spark tests with Gluten enabled. For the newly added tests here,
if they aren't variants of existing tests in the parent Spark test class, it
may be better to move them to a dedicated test suite. Could you do a refactor?
##########
gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala:
##########
@@ -1027,7 +1027,7 @@ class VeloxTestSettings extends BackendTestSettings {
.exclude("SPARK-49386: test SortMergeJoin (with spill by size threshold)")
enableSuite[GlutenMathFunctionsSuite]
// TODO: fix on Spark-4.1 see https://github.com/apache/spark/pull/50230
Review Comment:
Should we remove this comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
