pavanreddy02 created SPARK-61866:
------------------------------------

             Summary: Vector validation in checkNonNanVectors is slow on JVMs 
that do not inline virtual calls
                 Key: SPARK-61866
                 URL: https://issues.apache.org/jira/browse/SPARK-61866
             Project: Spark
          Issue Type: Improvement
          Components: ML, MLlib
    Affects Versions: 4.0.5
            Reporter: pavanreddy02


SPARK-45547 replaced the compiled validateVector UDF in 
DatasetUtils.checkNonNanVectors with the Catalyst higher-order function 
exists(). ArrayExists extends CodegenFallback, so its lambda is interpreted per 
element, costing several virtual calls per element of every vector. HotSpot 
inlines these calls so no regression was visible there, but on JVMs that do not 
(e.g. OpenJ9 on s390x) the cost is proportional to the vector size on every 
row, for every algorithm that validates its features. Proposal: restore the 
compiled UDF in checkNonNanVectors only. The VectorSizeHint and NaiveBayes 
changes from SPARK-45547 are untouched. Benchmark details will be added as 
comments.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to