zhengruifeng opened a new pull request, #58626: URL: https://github.com/apache/spark/pull/58626
### What changes were proposed in this pull request? This PR adds an internal `VectorDotProduct` Catalyst expression, registered as `ml_vector_dot_product`, for MLlib dense and sparse vectors. The expression supports interpreted evaluation and whole-stage code generation for all dense/sparse input combinations. It uses the expression in `LinearRegressionModel` and binomial `LogisticRegressionModel` transforms. Multinomial logistic regression remains unchanged. ### Why are the changes needed? Linear regression and binomial logistic regression currently compute vector dot products in Scala UDFs. This requires converting the vector UDT to MLlib objects for every row and prevents Catalyst from generating the dot-product loop as part of the query. The internal expression operates directly on the vector's SQL struct representation, preserves sparse coefficients without densifying them, and allows Catalyst code generation. ### Does this PR introduce _any_ user-facing change? No. This changes the internal execution of ML model transforms. ### How was this patch tested? New tests cover interpreted and code-generated evaluation for dense/dense, dense/sparse, sparse/dense, and sparse/sparse inputs, as well as nulls and mismatched dimensions. The following focused suites and cases were run: ```text build/sbt 'catalyst/testOnly *VectorExpressionsSuite' build/sbt 'mllib/testOnly *FunctionsSuite -- -z "vector_dot_product"' build/sbt 'mllib/testOnly *LinearRegressionSuite -- -z "can transform data with LinearRegressionModel"' build/sbt 'mllib/testOnly *LogisticRegressionSuite -- -z "binary logistic regression: Predictor, Classifier methods"' ``` `dev/scalastyle` also passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
