zhengruifeng opened a new pull request, #58848: URL: https://github.com/apache/spark/pull/58848
### What changes were proposed in this pull request? This PR initializes every active class slice of the reused multinomial logistic aggregation buffer with its margin offset before calling GEMM with `beta = 1.0`. It also adds regression coverage that compares a single block with multiple dense and sparse blocks. The second block is smaller, and one intercept is zero. ### Why are the changes needed? `MultinomialLogisticBlockAggregator` reuses its margin buffer across blocks. Previously, slices whose offset was zero were not cleared. They could therefore retain multipliers produced while processing the previous block. GEMM then added the next block's margins to those stale values, producing incorrect loss and gradient results. See https://issues.apache.org/jira/browse/SPARK-59562. ### Does this PR introduce _any_ user-facing change? Yes. Multinomial logistic regression now computes the correct loss and gradient when an aggregator reuses its buffer across blocks and a class has a zero margin offset. ### How was this patch tested? Added dense and sparse multi-block regression coverage and ran: ``` JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 ./build/sbt \ 'mllib/testOnly org.apache.spark.ml.optim.aggregator.MultinomialLogisticBlockAggregatorSuite' ``` All 9 tests passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
