This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 90a284856c14 [SPARK-57957][ML][TEST] Deflake GaussianMixtureSuite 'GMM
support instance weighting' on macOS using well-posed data
90a284856c14 is described below
commit 90a284856c14f53490adca3266a4aaf1dc8b7f11
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Jul 6 18:51:35 2026 +0900
[SPARK-57957][ML][TEST] Deflake GaussianMixtureSuite 'GMM support instance
weighting' on macOS using well-posed data
### What changes were proposed in this pull request?
Change the `GMM support instance weighting` test to fit `rDataset` with
`k=2` instead of fitting `k=5` on `KMeansSuite.generateKMeansData(50, 3, 5)`.
### Why are the changes needed?
The generated KMeans data is 5 clusters of identical points (zero
within-cluster variance). Fitting k=5 Gaussians makes the covariances singular
and the EM fit ill-posed, so the uniform-weighted and unweighted fits converge
to different component-collapse patterns. On the macOS-26 runner
(`build_maven_java21_macos26`) this deterministically fails the mixture-weight
comparison (0.0197 vs 0.1047).
Reducing the instance weight (as in SPARK-37317, which reduced 100->90) and
increasing maxIter were both verified on a macOS-26 runner to NOT help —
increasing maxIter actually makes a component collapse further (7.6e-11 vs
0.116). Running the same `unweighted == uniform-weighted` invariant on
well-posed data (`rDataset`, real variance, `k=2` — already used stably by
another test in this suite) makes both fits converge to the same optimum.
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
`GaussianMixtureSuite` on a macOS-26 GitHub Actions runner.
- **Before (failing on `apache/spark` macOS-26,
`build_maven_java21_macos26`):**
https://github.com/apache/spark/actions/runs/28753698265/job/85259893661 — `GMM
support instance weighting *** FAILED ***`, `Expected 0.01972564065075309 and
0.10476714410584831 to be within 0.001`.
- **After (passing with this change, macOS-26 runner):**
https://github.com/HyukjinKwon/spark-agent6/actions/runs/28773420633/job/85311928908
— `GaussianMixtureSuite: Tests: succeeded 13, failed 0`, `All tests passed.`
Also passes on Linux.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
This pull request and its description were written by Isaac.
Closes #57035 from HyukjinKwon/ci-fix/agent5-gmm-macos-wellposed.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 0a6bbee26f7d370b0800b7784c40c672bd917148)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../org/apache/spark/ml/clustering/GaussianMixtureSuite.scala | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/clustering/GaussianMixtureSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/clustering/GaussianMixtureSuite.scala
index c8a748e25139..93064b8440be 100644
---
a/mllib/src/test/scala/org/apache/spark/ml/clustering/GaussianMixtureSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/ml/clustering/GaussianMixtureSuite.scala
@@ -270,12 +270,12 @@ class GaussianMixtureSuite extends MLTest with
DefaultReadWriteTest {
}
test("GMM support instance weighting") {
- val gm1 = new GaussianMixture().setK(k).setMaxIter(20).setSeed(seed)
- val gm2 = new
GaussianMixture().setK(k).setMaxIter(20).setSeed(seed).setWeightCol("weight")
+ val gm1 = new GaussianMixture().setK(2).setMaxIter(20).setSeed(seed)
+ val gm2 = new
GaussianMixture().setK(2).setMaxIter(20).setSeed(seed).setWeightCol("weight")
Seq(1.0, 10.0, 90.0).foreach { w =>
- val gmm1 = gm1.fit(dataset)
- val ds2 = dataset.select(col("features"), lit(w).as("weight"))
+ val gmm1 = gm1.fit(rDataset)
+ val ds2 = rDataset.select(col("features"), lit(w).as("weight"))
val gmm2 = gm2.fit(ds2)
modelEquals(gmm1, gmm2)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
