masahi edited a comment on pull request #9261: URL: https://github.com/apache/tvm/pull/9261#issuecomment-942895664
I think it is reasonable to ask users for an estimate of a range of dynamism. Without such estimate, or if the estimate turns out to be off, we just fallback to a default kernel that works on any input shapes, which shouldn't be too slow anyway since it is cutlass, not TVM :) This should be a reasonable trade off between performance and convenience of dynamic models: if they want to run fast on dynamic models, they need to help us, otherwise we cannot guarantee good performance. I believe even if a network contains many dynamic workloads, there is only one underlying dynamism, from which other dynamic shapes can be derived. For example, MaskRCNN contains some dynamic batch dense, conv2d and conv2d transpose, but all dynamic dims directly correspond to the number of boxes detected, or some numbers derived from it. For transformers, I guess the true dynamism is only in sequence length. So I think it wouldn't be too much of an ask to give an estimate of such numbers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
