[
https://issues.apache.org/jira/browse/SPARK-59618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng resolved SPARK-59618.
-----------------------------------
Fix Version/s: 4.2.2
4.1.5
4.3.1
4.4.0
Resolution: Fixed
Issue resolved by pull request 58890
[https://github.com/apache/spark/pull/58890]
> Concurrent F2J LAPACK initialization can hang MultivariateGaussian
> ------------------------------------------------------------------
>
> Key: SPARK-59618
> URL: https://issues.apache.org/jira/browse/SPARK-59618
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 4.2.0
> Reporter: Ruifeng Zheng
> Assignee: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.2, 4.1.5, 4.3.1, 4.4.0
>
>
> F2J LAPACK caches machine parameters in unsynchronized mutable static fields.
> During concurrent first use, Dlamc2 publishes first=false before lrmin and
> lrmax are initialized. A racing Dsyev call can consume zero machine limits
> and enter a non-terminating Dlascl scaling loop.
> This can occur when MultivariateGaussian instances evaluate their lazy
> covariance constants concurrently. Repeated thread dumps show one task
> remaining RUNNABLE in Dlascl -> Dsyev -> F2jLAPACK while its peer finishes
> normally.
> The proposed fix warms up the selected LAPACK backend once per executor JVM
> before concurrent eigSym calls. It does not change library dependencies or
> serialize real eigendecompositions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
