DickJC123 opened a new pull request #8313: Ci test randomness URL: https://github.com/apache/incubator-mxnet/pull/8313 This PR proposes two new features to how CI uses random data for testing: 1. A 'with np_random_seed(NNN)' syntax to allow a single test to run with a set seed without forcing determinism on other tests in the same file, and 2. A set_np_random_seed() util function to set the seed randomly and output that setting to the nosetests log file. This permits a failure seen in a log file to be reproduced exactly. Numerous individual tests have been also made more robust. Most often the failures were the result of the finite difference method gradient not matching the symbol's gradient at a point when the gradient was either large (e.g. cube-root) or discontinuous (e.g. max). In other cases, tolerances needed to be added or increased. The proposed new approach is demonstrated on a simple test file of three tests. Assuming that the second test needs a set seed for robustness, the file might appear as: `def test_op1(): <op1 test> def test_op2(): np.random.seed(1234) <op2 test> def test_op3(): <op3 test> ` Even though test_op3() is OK with nondeterministic data, it will have only a single dataset because it is run after test_op2, which sets the seed. Also, if test_op1() were to fail, there would be no way to reproduce the failure, except for running the test individually to produce a new and hopefully similar failure. With the proposed approach, the test file becomes: `set_np_random_seed() def test_op1(): <op1 test> def test_op2(): with np_random_seed(1234): <op2 test> def test_op3(): <op3 test> ` set_np_random_seed() is in the global region of the test file and will set the seed differently for each run. The value set will appear in the nosetests output. The body of the test_op2 test now appears after the 'with np_random_seed(1234):' statement. This has the effect of setting the seed before the test runs, then reinstating the rng state after the test is over (failed or passed). Thus test_op3 will run with non-deterministic data. Finally, if say test_op1() fails, the seed is known so the failure can be reproduced. Further use of np.random.get_state() and np.random.set_state() can result in a modified test file that can be invoked as 'nosetests --verbose -s test_file.py:test_op1'. ## Checklist ## ### Essentials ### - [X ] Passed code style checking (`make lint`) - [X ] Changes are complete (i.e. I finished coding on this PR) - [X ] All changes have test coverage - [N/A ] For user-facing API changes, API doc string has been updated. - [X ] To my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change ### Changes ### - [ ] Feature1, tests, (and when applicable, API doc) - [ ] Feature2, tests, (and when applicable, API doc) ## Comments ## - If this change is a backward incompatible change, why must this change be made. - Intersting edge cases to note here ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
With regards, Apache Git Services
