[GitHub] [incubator-mxnet] KellenSunderland commented on a change in pull request #15399: Add unit tests for TensorRT integration and fix some bugs
KellenSunderland commented on a change in pull request #15399: Add unit tests
for TensorRT integration and fix some bugs
URL: https://github.com/apache/incubator-mxnet/pull/15399#discussion_r311301656
##
File path: src/operator/subgraph/tensorrt/tensorrt-inl.h
##
@@ -180,6 +253,17 @@ class TensorrtProperty : public SubgraphProperty {
n->attrs.name = "TensorRT" + std::to_string(subgraph_id);
n->attrs.op = Op::Get("_TensorRT");
CHECK(n->attrs.op);
+// prevent using Gamma value if using fix_gamma on BatchNorm
+DFSVisit(new_sym.outputs, [](const nnvm::NodePtr& node) {
+ if (node->op() == Op::Get("BatchNorm")) {
Review comment:
I think this one should be handled upstream. Have a look at
https://github.com/apache/incubator-mxnet/commit/6b22aa455e562db0e2744e94dc6a583191081b05
and let us know if you'd rather use this fix or if ours looks ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] KellenSunderland commented on a change in pull request #15399: Add unit tests for TensorRT integration and fix some bugs
KellenSunderland commented on a change in pull request #15399: Add unit tests
for TensorRT integration and fix some bugs
URL: https://github.com/apache/incubator-mxnet/pull/15399#discussion_r311301113
##
File path: src/operator/subgraph/tensorrt/tensorrt-inl.h
##
@@ -109,13 +111,70 @@ class TensorrtSelector : public SubgraphSelector {
bool isTRTCompatible(const nnvm::Node ) {
const std::string op_name = n.op()->name;
+if (op_name == "FullyConnected") {
+ const auto& param = nnvm::get(n.attrs.parsed);
+ return !param.no_bias;
+}
+
if (op_name == "Pooling") {
- return (n.attrs.dict.at("pool_type") == "avg" ||
- n.attrs.dict.at("pool_type") == "max");
+ const auto& param = nnvm::get(n.attrs.parsed);
+ if (param.layout.has_value()) {
+if (param.layout.value() == mshadow::kNHWC) {
+ LOG(INFO) << "Warning: NHWC layout (node: " << n.attrs.name
+<< ") is not supported by TensorRT";
+ return false;
+} else if (param.layout.value() == mshadow::kNDHWC) {
+ LOG(INFO) << "Warning: NDHWC layout (node: " << n.attrs.name
+<< ") is not supported by TensorRT";
+ return false;
+}
+ }
+ if (param.pooling_convention != pool_enum::kValid && !param.global_pool)
+return false;
+ if (param.pool_type == pool_enum::kAvgPooling) {
+if ((!param.global_pool) &&
+(!param.count_include_pad.has_value() ||
param.count_include_pad.value()))
+ return false;
+return true;
+ } else if (param.pool_type == pool_enum::kMaxPooling) {
+return true;
+ } else {
+return false;
+ }
}
-if (unconditionalTRTops.count(op_name)) {
- return true;
+if (op_name == "Convolution") {
+ const auto& param = nnvm::get(n.attrs.parsed);
+ if (!param.layout.has_value())
+return true;
+ switch (param.layout.value()) {
+case mshadow::kNCHW:
+case mshadow::kNCW:
+case mshadow::kNCDHW:
+ return true;
+case mshadow::kNHWC:
+ LOG(INFO) << "Warning: NHWC layout (node: " << n.attrs.name
+<< ") is not supported by TensorRT";
+ return false;
+case mshadow::kNDHWC:
+ LOG(INFO) << "Warning: NDHWC layout (node: " << n.attrs.name
+<< ") is not supported by TensorRT";
+ return false;
+default:
+ LOG(INFO) << "Warning: Layout (node: " << n.attrs.name
+<< ") is unknown (so unsupported by TensorRT)";
+ return false;
+ }
+}
+
+if (op_name == "Concat") {
+ const auto& param = nnvm::get(n.attrs.parsed);
+ return (param.dim != 0);
+}
+
+if (op_name == "Dropout") {
Review comment:
Ok, non-blocking comment for this PR. I'm just thinking about adding a
warning in the future if people are using TRT with operations that don't make
sense at inference time (Dropout, Ident, Empty Concats or Copies, etc.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] KellenSunderland commented on a change in pull request #15399: Add unit tests for TensorRT integration and fix some bugs
KellenSunderland commented on a change in pull request #15399: Add unit tests
for TensorRT integration and fix some bugs
URL: https://github.com/apache/incubator-mxnet/pull/15399#discussion_r302338157
##
File path: src/operator/subgraph/tensorrt/tensorrt-inl.h
##
@@ -180,6 +253,17 @@ class TensorrtProperty : public SubgraphProperty {
n->attrs.name = "TensorRT" + std::to_string(subgraph_id);
n->attrs.op = Op::Get("_TensorRT");
CHECK(n->attrs.op);
+// prevent using Gamma value if using fix_gamma on BatchNorm
+DFSVisit(new_sym.outputs, [](const nnvm::NodePtr& node) {
+ if (node->op() == Op::Get("BatchNorm")) {
Review comment:
Why not just check for FixGamma = true during nnvm -> Onnx conversion and
set gamma to 1 if it's true?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] KellenSunderland commented on a change in pull request #15399: Add unit tests for TensorRT integration and fix some bugs
KellenSunderland commented on a change in pull request #15399: Add unit tests for TensorRT integration and fix some bugs URL: https://github.com/apache/incubator-mxnet/pull/15399#discussion_r302338438 ## File path: tests/python/gpu/test_tensorrt.py ## @@ -0,0 +1,437 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review comment: Super helpful tests, thanks Clement. It's ok to merge in a subset of these at a time, but I'd try to avoid pushing commented out code / too many todos upstream. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] KellenSunderland commented on a change in pull request #15399: Add unit tests for TensorRT integration and fix some bugs
KellenSunderland commented on a change in pull request #15399: Add unit tests
for TensorRT integration and fix some bugs
URL: https://github.com/apache/incubator-mxnet/pull/15399#discussion_r302338157
##
File path: src/operator/subgraph/tensorrt/tensorrt-inl.h
##
@@ -180,6 +253,17 @@ class TensorrtProperty : public SubgraphProperty {
n->attrs.name = "TensorRT" + std::to_string(subgraph_id);
n->attrs.op = Op::Get("_TensorRT");
CHECK(n->attrs.op);
+// prevent using Gamma value if using fix_gamma on BatchNorm
+DFSVisit(new_sym.outputs, [](const nnvm::NodePtr& node) {
+ if (node->op() == Op::Get("BatchNorm")) {
Review comment:
Why not just check for FixGamma = true during nnvm -> Onnx conversion and
set gamma to 0 if it's true?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] KellenSunderland commented on a change in pull request #15399: Add unit tests for TensorRT integration and fix some bugs
KellenSunderland commented on a change in pull request #15399: Add unit tests
for TensorRT integration and fix some bugs
URL: https://github.com/apache/incubator-mxnet/pull/15399#discussion_r302337630
##
File path: src/operator/subgraph/tensorrt/tensorrt-inl.h
##
@@ -109,13 +111,70 @@ class TensorrtSelector : public SubgraphSelector {
bool isTRTCompatible(const nnvm::Node ) {
const std::string op_name = n.op()->name;
+if (op_name == "FullyConnected") {
+ const auto& param = nnvm::get(n.attrs.parsed);
+ return !param.no_bias;
+}
+
if (op_name == "Pooling") {
- return (n.attrs.dict.at("pool_type") == "avg" ||
- n.attrs.dict.at("pool_type") == "max");
+ const auto& param = nnvm::get(n.attrs.parsed);
+ if (param.layout.has_value()) {
+if (param.layout.value() == mshadow::kNHWC) {
+ LOG(INFO) << "Warning: NHWC layout (node: " << n.attrs.name
+<< ") is not supported by TensorRT";
+ return false;
+} else if (param.layout.value() == mshadow::kNDHWC) {
+ LOG(INFO) << "Warning: NDHWC layout (node: " << n.attrs.name
+<< ") is not supported by TensorRT";
+ return false;
+}
+ }
+ if (param.pooling_convention != pool_enum::kValid && !param.global_pool)
+return false;
+ if (param.pool_type == pool_enum::kAvgPooling) {
+if ((!param.global_pool) &&
+(!param.count_include_pad.has_value() ||
param.count_include_pad.value()))
+ return false;
+return true;
+ } else if (param.pool_type == pool_enum::kMaxPooling) {
+return true;
+ } else {
+return false;
+ }
}
-if (unconditionalTRTops.count(op_name)) {
- return true;
+if (op_name == "Convolution") {
+ const auto& param = nnvm::get(n.attrs.parsed);
+ if (!param.layout.has_value())
+return true;
+ switch (param.layout.value()) {
+case mshadow::kNCHW:
+case mshadow::kNCW:
+case mshadow::kNCDHW:
+ return true;
+case mshadow::kNHWC:
+ LOG(INFO) << "Warning: NHWC layout (node: " << n.attrs.name
+<< ") is not supported by TensorRT";
+ return false;
+case mshadow::kNDHWC:
+ LOG(INFO) << "Warning: NDHWC layout (node: " << n.attrs.name
+<< ") is not supported by TensorRT";
+ return false;
+default:
+ LOG(INFO) << "Warning: Layout (node: " << n.attrs.name
+<< ") is unknown (so unsupported by TensorRT)";
+ return false;
+ }
+}
+
+if (op_name == "Concat") {
+ const auto& param = nnvm::get(n.attrs.parsed);
+ return (param.dim != 0);
+}
+
+if (op_name == "Dropout") {
Review comment:
Again, will TensorRT optimize this out? We don't want it at inference time
right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] KellenSunderland commented on a change in pull request #15399: Add unit tests for TensorRT integration and fix some bugs
KellenSunderland commented on a change in pull request #15399: Add unit tests for TensorRT integration and fix some bugs URL: https://github.com/apache/incubator-mxnet/pull/15399#discussion_r302336971 ## File path: src/operator/subgraph/tensorrt/nnvm_to_onnx.cc ## @@ -157,6 +157,12 @@ std::string ConvertNnvmGraphToOnnx( return serialized_onnx_graph; } +void ConvertIdentity(NodeProto* node_proto, const NodeAttrs& attrs, Review comment: Any idea if TRT actually optimizes this out? I've seen this in a few prod services :-/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
