AndrewZhaoLuo commented on code in PR #13074: URL: https://github.com/apache/tvm/pull/13074#discussion_r996157034
########## python/tvm/relay/frontend/onnx.py: ########## @@ -944,6 +946,36 @@ def _impl_v1(cls, inputs, attr, params): return Gelu._impl_v1([inp], attr, params) +class LayerNormalization(OnnxOpConverter): Review Comment: Do the tests themselves test for these outputs? In the spec they are listed as optional LayerNorm, much like BatchNorm has different behavior between training and inference time. As ONNX frontend is pretty much for inference, I would say it's ok to drop the other two outputs if it just simplifies the code a lot. ########## python/tvm/relay/frontend/onnx.py: ########## @@ -944,6 +946,36 @@ def _impl_v1(cls, inputs, attr, params): return Gelu._impl_v1([inp], attr, params) +class LayerNormalization(OnnxOpConverter): + """Operator converter for LayerNormalization from Microsoft onnxruntime contrib opset.""" + + @classmethod + def _impl_v17(cls, inputs, attr, params): + x = inputs[0] + gamma = inputs[1] + beta = inputs[2] + axis = attr.get("axis", -1) + eps = attr.get("epsilon", 1e-5) + # according to the onnx doc, given the int axis (default -1) + # to compute the mean and inv_stdev which are of dim [d[0], ..., d[axis-1], 1, ..., 1] + # the actual computation is over (axis, ..., rank(x) - 1) axes + # see https://github.com/onnx/onnx/blob/main/docs/Changelog.md#layernormalization-17 + rank = len(infer_shape(x)) + axis = tuple(range(axis, rank)) if axis >= 0 else tuple(range(rank + axis, rank)) + dtype = infer_type(x).checked_type.dtype + mean = _op.mean(x, axis, keepdims=True) Review Comment: I believe this is inefficient as mean will do a reduction along the axis and variance will do two reductions (one to find mean, one to find the mean of (X - E[X])^2. Giving three total reductions across the axis. Instead just manually do E[(X - E[X])^2] to get two total reductions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org