AndrewZhaoLuo commented on code in PR #13074:
URL: https://github.com/apache/tvm/pull/13074#discussion_r996196092
##########
python/tvm/relay/frontend/onnx.py:
##########
@@ -944,6 +946,36 @@ def _impl_v1(cls, inputs, attr, params):
return Gelu._impl_v1([inp], attr, params)
+class LayerNormalization(OnnxOpConverter):
+ """Operator converter for LayerNormalization from Microsoft onnxruntime
contrib opset."""
+
+ @classmethod
+ def _impl_v17(cls, inputs, attr, params):
+ x = inputs[0]
+ gamma = inputs[1]
+ beta = inputs[2]
+ axis = attr.get("axis", -1)
+ eps = attr.get("epsilon", 1e-5)
+ # according to the onnx doc, given the int axis (default -1)
+ # to compute the mean and inv_stdev which are of dim [d[0], ...,
d[axis-1], 1, ..., 1]
+ # the actual computation is over (axis, ..., rank(x) - 1) axes
+ # see
https://github.com/onnx/onnx/blob/main/docs/Changelog.md#layernormalization-17
+ rank = len(infer_shape(x))
+ axis = tuple(range(axis, rank)) if axis >= 0 else tuple(range(rank +
axis, rank))
+ dtype = infer_type(x).checked_type.dtype
+ mean = _op.mean(x, axis, keepdims=True)
Review Comment:
Yes, common subexpression elimination pass should eliminate this. However,
CSE can't do magic, and has limitations. You can probably check if it will be
removed here by running this generated relay through the pass. Perhaps it will.
In general, unless it adds a great amount of overhead in the reasoning of a
piece of code, I would not duplicate ops however.
There are cases where CSE should be a-okay but does not:
https://github.com/apache/tvm/issues/12977
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
