zhengruifeng commented on a change in pull request #25178: [SPARK-28421][ML] SparseVector.apply performance optimization URL: https://github.com/apache/spark/pull/25178#discussion_r305176094
########## File path: mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala ########## @@ -603,6 +603,19 @@ class SparseVector @Since("2.0.0") ( private[spark] override def asBreeze: BV[Double] = new BSV[Double](indices, values, size) + override def apply(i: Int): Double = { + if (i < 0 || i >= size) { + throw new IndexOutOfBoundsException(s"Index $i out of bounds [0, $size)") + } + + if (indices.isEmpty || i < indices(0) || i > indices(indices.length - 1)) { Review comment: you can see that if the `nnz` grows, the speed up decrese. That is because with a big `nnz`, the searching complexity `log(nnz)` dominate the whole process. However, when `nnz` is a small number (most frequently), the conversion is relatively the main part. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org