szha opened a new pull request #13896: [WIP] Cudnn dropout URL: https://github.com/apache/incubator-mxnet/pull/13896 ## Description ## Use dropout in CuDNN Tested on p3.2x (V100). Test case: ```python import mxnet as mx a = mx.nd.ones((10, 200, 300, 500), ctx=mx.gpu(0)) a.attach_grad() mx.autograd.set_recording(True) %timeit mx.nd.Dropout(a, 0.5, mode='always').wait_to_read() ``` | | fwd | fwd+bwd | |----------|-------|---------| | GPU | 46ms | 48ms | | prev GPU | 305ms | 321ms | ## Checklist ## ### Essentials ### Please feel free to remove inapplicable items for your PR. - [x] Changes are complete (i.e. I finished coding on this PR) - [x] All changes have test coverage: - Unit tests are added for small changes to verify correctness (e.g. adding a new operator) - Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore) - Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL) - [x] Code is well-documented: - For new C++ functions in header files, their functionalities and arguments are documented. - Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html - [x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change ### Changes ### - [x] CuDNN Dropout ## Comments ## - `cudnnSetDropoutDescriptor` is an expensive call due to initialization on each of the stream multiprocessor on a GPU. Since cudnn 7, `cudnnRestoreDropoutDescriptor` becomes available so that the initialized space can be cached. This descriptor is currently used in both Dropout op and RNN op. We need a mechanism for caching this so that initialization on each stream happens only once, as the same desc can be shared among operators.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
