kpuatamazon commented on issue #17559: [MXNET-1446] Quantization: intgemm matrix multiply wrappers URL: https://github.com/apache/incubator-mxnet/pull/17559#issuecomment-590265641 @ciyongch Here is a usage example. ```python import mxnet as mx #This is done offline. weight = mx.nd.random_uniform(shape=(8,64), low=-1.0, high=1.0) weight_max = mx.nd.contrib.intgemm_maxabsolute(weight) weight_prepared = mx.nd.contrib.intgemm_prepare_weight(weight, weight_max) data = mx.nd.random_uniform(shape=(1,64), low=-1.0, high=1.0) #Fused multiply quantizes on the fly. product1 = mx.nd.contrib.intgemm_fully_connected(data, weight_prepared, scaling = weight_max / 127.0, num_hidden = 8, no_bias = True) #One can also have quantized data. data_max = mx.nd.contrib.intgemm_maxabsolute(data) data_prepared = mx.nd.contrib.intgemm_prepare_data(data, data_max) product2 = mx.nd.contrib.intgemm_fully_connected(data_prepared, weight_prepared, scaling = weight_max / 127.0 * data_max / 127.0, num_hidden = 8, no_bias = True) baseline = mx.nd.FullyConnected(data, weight, num_hidden = 8, no_bias = True) ``` The `prepare_data` step is just a quantizer. The `prepare_weight` step does some element rearragement into a CPU-dependent format so the API isn't quite the same as the current one. And bias works in the usual way. Internally intgemm takes a template argument for what to do with the int32 output while it's still in a register. So one can write out floats or apply activation functions in registers first. The multiple of 8 issue is relatively easy to fix; that was just us being lazy about smaller tiles. Having inner dimension a multiple of 64 is rather useful for alignment purposes; how does MXNet deal with padding?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
