This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 487a40b3dc3b [SPARK-57951][ML][TEST] Tolerate last-ULP FP differences
in MLTest single-prediction checks for macOS
487a40b3dc3b is described below
commit 487a40b3dc3bedfcc99b477662945aa5c16beb85
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Jul 6 17:29:45 2026 +0900
[SPARK-57951][ML][TEST] Tolerate last-ULP FP differences in MLTest
single-prediction checks for macOS
### What changes were proposed in this pull request?
`MLTest.testClassificationModelSingleRawPrediction` and
`testProbClassificationModelSingleProbPrediction` compared the DataFrame
`transform`
output against the scalar `predictRaw` / `predictProbability` output with
exact `===`
equality. This PR compares the two prediction vectors with a tight absolute
tolerance
(`1e-9`) via a small `assertVectorsAlmostEqual` helper instead.
### Why are the changes needed?
On arm64 macOS the DataFrame path and the scalar path can round differently
in the last
ULP (e.g. `1.543502002724983` vs `1.5435020027249835`), so the exact `===`
check fails:
```
[1.543502002724983,-1.543502002724983] did not equal
[1.5435020027249835,-1.5435020027249835] (MLTest.scala:199)
```
This fails suites such as `MultilayerPerceptronClassifierSuite` on the
scheduled
`Build / Maven (Scala 2.13, JDK 21, MacOS-26)` lane, while the same tests
are bit-identical
on Linux. A `1e-9` absolute tolerance absorbs last-ULP platform rounding
while still catching
any real discrepancy between the two code paths.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
Ran the affected classifier suites on a `macos-15` GitHub Actions runner
(arm64, same class as
`macos-26`):
- `MultilayerPerceptronClassifierSuite`, `LogisticRegressionSuite`,
`LinearSVCSuite`,
`NaiveBayesSuite` — **113 tests, 0 failures** (the `prediction on single
instance` cases that
previously failed now pass).
Passed: https://github.com/HyukjinKwon/spark/actions/runs/28766568832
Before this change the same suites fail on `macos-26`:
https://github.com/apache/spark/actions/runs/28753698265
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
Closes #57032 from HyukjinKwon/ci-fix/agent4-mllib-fp-tol-pr.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit fa81bf5a1dd3ec955a18186ff8038fb5a234d02d)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../scala/org/apache/spark/ml/util/MLTest.scala | 23 ++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/mllib/src/test/scala/org/apache/spark/ml/util/MLTest.scala
b/mllib/src/test/scala/org/apache/spark/ml/util/MLTest.scala
index a35b19b2816e..23d27162654c 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/util/MLTest.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/util/MLTest.scala
@@ -196,7 +196,11 @@ trait MLTest extends StreamTest with TempDirectory { self:
Suite =>
model.transform(dataset).select(model.getFeaturesCol,
model.getRawPredictionCol)
.collect().foreach {
case Row(features: Vector, rawPrediction: Vector) =>
- assert(rawPrediction === model.predictRaw(features))
+ // Compare with a tight tolerance rather than exact equality: the
DataFrame
+ // `transform` path and the scalar `predictRaw` path can round
differently in the
+ // last ULP on some platforms (e.g. arm64 macOS, where native BLAS
diverges from
+ // JVM scalar math), which broke the MacOS-26 lane while Linux stayed
bit-identical.
+ assertVectorsAlmostEqual(rawPrediction, model.predictRaw(features))
}
}
@@ -206,7 +210,22 @@ trait MLTest extends StreamTest with TempDirectory { self:
Suite =>
model.transform(dataset).select(model.getFeaturesCol,
model.getProbabilityCol)
.collect().foreach {
case Row(features: Vector, probPrediction: Vector) =>
- assert(probPrediction === model.predictProbability(features))
+ assertVectorsAlmostEqual(probPrediction,
model.predictProbability(features))
+ }
+ }
+
+ /**
+ * Asserts two prediction vectors are equal up to a tight absolute
tolerance. Used instead of
+ * exact `===` so that last-ULP rounding differences between the DataFrame
`transform` path and
+ * the scalar `predict*` path (observed on arm64 macOS) do not fail the
tests, while any real
+ * discrepancy is still caught.
+ */
+ private def assertVectorsAlmostEqual(actual: Vector, expected: Vector): Unit
= {
+ assert(actual.size === expected.size,
+ s"vector sizes differ: ${actual.size} vs ${expected.size}")
+ actual.toArray.zip(expected.toArray).zipWithIndex.foreach { case ((a, e),
i) =>
+ assert(math.abs(a - e) <= 1e-9,
+ s"prediction vectors differ at index $i: $a vs $e (actual=$actual,
expected=$expected)")
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
