Hello.

Le jeu. 16 mai 2019 à 12:06, Alex Herbert <alex.d.herb...@gmail.com> a écrit :
>
> I have run the stress test using the new application. The new application has 
> two major changes over the previous application:
>
> 1. It detects the platform byte-order and sends the bits in the correct order 
> to be read by a C application
> 2. The bridge to TestU01 has been updated to use all the input int values, 
> previously it was using every other int value
>
> So we can expect differences from both test suites Dieharder and TestU01 
> BigCrush.
>
> For reference here are the old results (from the user guide, reordered to the 
> RandomSource enum order):
>
> RNG                     Dieharder       TestU01 (BigCrush)
> JDK                     11, 12, 13      74, 72, 75
> WELL_512_A              0, 0, 0         7, 6, 6
> WELL_1024_A             0, 0, 0         4, 4, 5
> WELL_19937_A            0, 0, 0         3, 2, 3
> WELL_19937_C            0, 1, 0         2, 2, 3
> WELL_44497_A            0, 0, 0         2, 3, 3
> WELL_44497_B            0, 0, 0         2, 2, 2
> MT                      0, 1, 0         3, 2, 2
> ISAAC                   0, 0, 1         0, 1, 0
> SPLIT_MIX_64            0, 0, 0         2, 0, 0
> XOR_SHIFT_1024_S        0, 0, 0         2, 0, 0
> TWO_CMRES               1, 1, 1         0, 0, 1
> MT_64                   0, 0, 1         3, 2, 3
> MWC_256                 0, 0, 0         0, 0, 0
> KISS                    0, 0, 0         1, 2, 0
>
> Here are the new results:
>
> RNG                     Dieharder       TestU01 (BigCrush)
> JDK                     4,4,4,4,4       74,72,74,73,74
> WELL_512_A              0,0,0,0,0       7,6,6,6,6
> WELL_1024_A             0,0,0,0,0       4,4,5,4,4
> WELL_19937_A            0,1,0,0,1       3,3,2,2,2
> WELL_19937_C            0,0,0,0,0       2,2,3,2,2
> WELL_44497_A            0,0,0,0,0       2,2,2,2,3
> WELL_44497_B            0,0,0,0,0       2,3,2,2,2
> MT                      0,0,0,0,0       2,3,2,2,2
> ISAAC                   0,0,0,0,0       0,1,2,0,0
> SPLIT_MIX_64            0,0,0,0,0       1,0,0,0,0
> XOR_SHIFT_1024_S        0,0,0,0,0       0,0,0,0,0
> TWO_CMRES               2,2,2,2,2       4,3,3,5,4
> MT_64                   0,0,0,0,0       2,3,2,2,2
> MWC_256                 0,1,0,0,0       0,0,0,2,0
> KISS                    0,0,0,0,0       0,0,0,0,0
> XOR_SHIFT_1024_S_PHI    0,0,0,0,0       0,0,0,0,0
> XO_RO_SHI_RO_64_S       0,0,0,0,0       1,1,2,1,3
> XO_RO_SHI_RO_64_SS      0,0,0,0,0       0,0,0,0,0
> XO_SHI_RO_128_PLUS      0,0,1,0,0       1,2,2,1,1
> XO_SHI_RO_128_SS        0,0,0,1,0       0,1,0,0,0
> XO_RO_SHI_RO_128_PLUS   0,0,0,0,0       0,1,0,0,0
> XO_RO_SHI_RO_128_SS     0,0,0,0,0       1,0,1,0,0
> XO_SHI_RO_256_PLUS      0,1,0,0,0       0,0,0,0,0
> XO_SHI_RO_256_SS        0,0,0,0,0       0,1,0,2,1
> XO_SHI_RO_512_PLUS      0,0,0,0,1       0,0,0,2,2
> XO_SHI_RO_512_SS        0,0,0,0,0       0,1,0,1,0
>
> (Note: All of the single fails except one under Dieharder are for the flawed 
> diehard_sums test. I include it here for direct comparison with old results. 
> I would recommend we strip this from the new results for the user guide.)
>
> I ran them 3 times. Then because the results were different (mainly for the 
> JDK generator for Dieharder) I doubled checked everything and ran another 2. 
> Results are still the same. Dieharder is much better for the JDK than 
> previously. It systematically fails:
>
> diehard_opso:0
> diehard_oqso:0
> diehard_dna:0
> dab_bytedistrib:0
>
> The TWO_CMRES generator is now worse as it is systematically failing:
>
> diehard_oqso:0
> diehard_dna:0
>
> The results from BigCrush are similar for JDK and all the others except 
> TWO_CMRES. This is now failing a few more tests. It systematically fails:
>
> 1  SerialOver, r = 0
> 41  Permutation, t = 5
> 42  Permutation, t = 7
>
> To check the JDK results for Dieharder I ran it 5 times using the wrong 
> platform byte order (i.e. what the previous test application was doing).
>
> Old results : 11, 12, 13
> New results: 11,16,14,14,15
>
> So this matches up. If the JDK output is byte reversed it is a poor generator.
>
> A few sources I have read indicate that BigCrush favours the upper bits of a 
> generator. A test should therefore run the generator bit reversed through the 
> test application. Here are the full forward and backward results ignoring the 
> Diehard sums test:
>
> RNG                     Bit-reversed    Dieharder       TestU01 (BigCrush)
> JDK                     false           4,4,4,4,4       74,72,74,73,74
> JDK                     true            42,42,43,49,49  35,34,35,36,36
> WELL_512_A              false           0,0,0,0,0       7,6,6,6,6
> WELL_512_A              true            0,0,1,0,0       7,6,6,7,6
> WELL_1024_A             false           0,0,0,0,0       4,4,5,4,4
> WELL_1024_A             true            0,0,0,0,0       4,4,4,4,4
> WELL_19937_A            false           0,1,0,0,0       3,3,2,2,2
> WELL_19937_A            true            0,0,0,0,0       3,2,2,2,3
> WELL_19937_C            false           0,0,0,0,0       2,2,3,2,2
> WELL_19937_C            true            0,0,0,0,0       3,2,2,3,2
> WELL_44497_A            false           0,0,0,0,0       2,2,2,2,3
> WELL_44497_A            true            0,0,0,0,0       3,3,3,2,2
> WELL_44497_B            false           0,0,0,0,0       2,3,2,2,2
> WELL_44497_B            true            0,0,0,0,0       2,2,2,2,3
> MT                      false           0,0,0,0,0       2,3,2,2,2
> MT                      true            0,0,0,0,0       2,2,3,3,3
> ISAAC                   false           0,0,0,0,0       0,1,2,0,0
> ISAAC                   true            0,0,0,0,0       0,0,0,0,0
> SPLIT_MIX_64            false           0,0,0,0,0       1,0,0,0,0
> SPLIT_MIX_64            true            0,0,0,0,0       0,1,0,0,0
> XOR_SHIFT_1024_S        false           0,0,0,0,0       0,0,0,0,0
> XOR_SHIFT_1024_S        true            0,0,0,0,0       0,0,1,0,0
> TWO_CMRES               false           2,2,2,2,2       4,3,3,5,4
> TWO_CMRES               true            7,5,5,7,6       4,3,4,4,4
> MT_64                   false           0,0,0,0,0       2,3,2,2,2
> MT_64                   true            0,0,0,0,0       2,2,2,2,2
> MWC_256                 false           0,0,0,0,0       0,0,0,2,0
> MWC_256                 true            0,0,0,0,0       1,0,0,0,0
> KISS                    false           0,0,0,0,0       0,0,0,0,0
> KISS                    true            0,0,0,0,0       0,0,1,0,1
> XOR_SHIFT_1024_S_PHI    false           0,0,0,0,0       0,0,0,0,0
> XOR_SHIFT_1024_S_PHI    true            0,0,0,0,0       0,0,2,0,0
> XO_RO_SHI_RO_64_S       false           0,0,0,0,0       1,1,2,1,3
> XO_RO_SHI_RO_64_S       true            0,0,0,0,0       2,2,2,2,2
> XO_RO_SHI_RO_64_SS      false           0,0,0,0,0       0,0,0,0,0
> XO_RO_SHI_RO_64_SS      true            0,0,0,0,0       1,0,0,0,0
> XO_SHI_RO_128_PLUS      false           0,0,0,0,0       1,2,2,1,1
> XO_SHI_RO_128_PLUS      true            0,0,0,0,0       2,2,2,2,2
> XO_SHI_RO_128_SS        false           0,0,0,0,0       0,1,0,0,0
> XO_SHI_RO_128_SS        true            0,0,0,0,0       0,0,0,0,0
> XO_RO_SHI_RO_128_PLUS   false           0,0,0,0,0       0,1,0,0,0
> XO_RO_SHI_RO_128_PLUS   true            0,0,0,0,0       2,1,1,1,2
> XO_RO_SHI_RO_128_SS     false           0,0,0,0,0       1,0,1,0,0
> XO_RO_SHI_RO_128_SS     true            0,0,0,0,0       0,0,2,0,0
> XO_SHI_RO_256_PLUS      false           0,0,0,0,0       0,0,0,0,0
> XO_SHI_RO_256_PLUS      true            0,0,0,0,0       0,0,0,0,0
> XO_SHI_RO_256_SS        false           0,0,0,0,0       0,1,0,2,1
> XO_SHI_RO_256_SS        true            0,0,0,0,0       0,1,1,1,2
> XO_SHI_RO_512_PLUS      false           0,0,0,0,0       0,0,0,2,2
> XO_SHI_RO_512_PLUS      true            0,0,0,0,0       1,0,0,0,1
> XO_SHI_RO_512_SS        false           0,0,0,0,0       0,1,0,1,0
> XO_SHI_RO_512_SS        true            0,0,0,0,0       0,1,1,0,0
>
> So bit reversed the JDK is terrible at Dieharder. It actually improves for 
> BigCrush from terrible to less terrible. TWO_CMRES is a bit worse when 
> bit-reversed at Dieharder but no different at BigCrush (it was already 
> systematically failing 3 tests).

Is it the same version of "BigCrush"?  I'm surprised that TWO_CMRES
have much more failures (bit-reversed or not).

>
> All the other generators have similar results when bit reversed. So adding 
> the bit-reversed results to the user-guide does not appear worthwhile. I will 
> archive these and they can be added later if required, for example to show a 
> good generator against a bad one. This will only be relevant if the library 
> adds reference implementations of bad generators.

It's on Abhishek's TODO list (e.g. "LCG").

> Currently only the JDK is bad generator.
>
> Next:
>
> I have added a ‘results' command to the stress test application that can 
> generate these results tables. It requires some header information not found 
> in the old results files so only works with the new results. It can generate 
> the APT table directly for the user guide. It will be useful going forward 
> when more generators are added to update the results.
>
> The new results are named using the test suite (dh_ or tu_), optionally the 
> bit-reversed flag (r_), the enum ordinal and the trial run:
>
> dh_1_1 = Dieharder for JDK trial 1
> tu_1_1 = BigCrush for JDK trial 1
> dh_r_2_3 = Dieharder bit reversed for WELL_512_A trial 3
>
> I propose to:
>
> - Delete all the old results and add these new ones using a new directory 
> structure. All results can reside in a single directory.
> - Ignore for now the bit-reversed results.
> - Delete the old stress test code. The new code supersedes all functionality 
> of the old version.
> - Commit the new ‘results’ command when I have confirmed the APT table is 
> correctly generated.

+1

>
> Questions:
>
> 1. Do we stick to using 3 trials or update to 5 (because I have the results)?

+1

> 2. Do we remove the diehard_sums test result?
>
> I would recommend removing diehard_sums. It pollutes the results for most 
> generators with a spurious fail that should be ignored. So I think we should 
> ignore it.

+0 (as you wish)

Gilles

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Reply via email to