Hi
I want to find matrix inverse of (XTranspose * X). PFB my code.
This code does not work for even slight larger dataset. Please help me if
the approach is correct.
val sqlQuery = "SELECT column1,column2 ,column3 FROM " + tableName
val matrixDF` = sqlContext.sql(sqlQuery)
var identityArray = Array.fill[String](numberOfRows)("1.0") // array
that contains 1 for calculating the intercept
val collectionString =
Array.ofDim[String](numberOfColumns,numberOfRows) // store the collection
as string
val independentCollectionDoubleType =
Array.ofDim[Double](numberOfColumns,numberOfRows) // store the collection
as double
//concatenate the values of all the as string type
for (i <- 0 to numberOfColumns - 1) {
val matrixStringDF = matrixDF.map(_.get(i).toString)
val matrixArray = matrixStringDF.collect()
identityArray = Array.concat(identityArray, matrixArray)
collectionString(i) = matrixArray
}
val valuesDouble = identityArray.map(value => value.toDouble) //Array
of all the values and identity as an array of double
val valuesMatrix = Matrices.dense(numberOfRows, numberOfColumns + 1,
valuesDouble) //matrix
val transposeMatrix = valuesMatrix.transpose // transpose of matrix
val transposeMatrix_matrix =
transposeMatrix.multiply(valuesMatrix.asInstanceOf[DenseMatrix]) //
transpose of matrix * matrix
val transposeMatrix_matrixAsArray = transposeMatrix_matrix.values //
pdt as array
val transposeMatrix_matrixBreeze = new
linalg.DenseMatrix(transposeMatrix_matrix.numRows,
transposeMatrix_matrix.numCols, transposeMatrix_matrixAsArray, 0) //breeze
dense matrix of the product
val inverse = linalg.inv(transposeMatrix_matrixBreeze)
--
Thanks and Regards
Arun
