[ https://issues.apache.org/jira/browse/SYSTEMML-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084943#comment-16084943 ]
Mike Dusenberry commented on SYSTEMML-1762: ------------------------------------------- cc [~mboehm7] > Improve the robustness of sparse matrix reshape function for the Spark mode > --------------------------------------------------------------------------- > > Key: SYSTEMML-1762 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1762 > Project: SystemML > Issue Type: Bug > Components: Algorithms, ParFor, Runtime > Reporter: Fei Hu > Assignee: Fei Hu > Attachments: MNIST_Distrib_Sgd.scala > > > When running the [distributed MNIST LeNet example | > https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet_distrib_sgd.dml], > it works well in the hybrid mode. But in the Spark mode, there is a > {{java.lang.NullPointerException}} error when reshaping the sparse matrix. > The involved function is > {{org.apache.sysml.runtime.matrix.data.LibMatrixReorg#reshapeSparse}} . The > reason is that the output matrix index computed by > {{org.apache.sysml.runtime.matrix.data.LibMatrixReorg#computeResultBlockIndex}} > does not exist in the {{HashMap<MatrixIndexes,MatrixBlock> rix}}. > To reproduce the error, the attached scala file {{MNIST_Distrib_Sgd.scala}} > could be used to run the distributed MNIST example. > In addition, if adding some codes to ignore the null output matrix block from > {{MatrixBlock out = rix.get(ixtmp)}}, the distributed MNIST example could > run in the Spark mode, but the result may not be right. -- This message was sent by Atlassian JIRA (v6.4.14#64029)