zheng-da commented on issue #9921: [DISCUSSION] module.contrib.SparseModule API URL: https://github.com/apache/incubator-mxnet/issues/9921#issuecomment-371185700 @eric-haibin-lin I agree the first benefit is important. I wonder if we can have forward() to pull required weights from kvstore automatically? to enable this, we'll need the operators that support sparse row weights to tell the executor which rows should be pulled. we probably need to add another attribute to NNVM for these operators. For the second benefit, I would argue that if you accept maintaining the entire weight matrix in each worker node, you can create an array lookup table (an array that maps row id to the physical location in the row sparse weight matrix) instead of a hashtable to reduce the lookup overhead. Maintaining such an array lookup table requires computation of O(nzr), the number of non-zero rows, for both cleaning the existing table and inserting entries for a new row sparse weight matrix. Maintaining the entire weight matrix in each worker node has similar overhead for updating rows. The lookup table has memory of O(n), instead of O(n * p), the size of the entire weight matrix.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
