It appears that each test case is being run twice. I think once to generate the xml output, and once to generate output for stdout and tell you whether it failed. The random number generator isn't being reseeded at the start of each test so they can produce different results, so you can sometimes see debug statements that indicate it should fail, but the output to stdout claims that is passes. I've attached a patch that make each test use it's own random number generating object which seems a tidier way to do this, and makes the tests repeatable.
Since there seem to be a fair number of choices of seed that produce an error fraction of over 0.2, I've also increased the acceptable error rate in the test to 0.3, as Alick initially suggested. On Sat, Apr 14, 2012 at 9:39 AM, Alick Zhao <alick9...@gmail.com> wrote: > On Sat, 14 Apr 2012 13:40:35 +0200, Martin Braun wrote: >> On Sat, Apr 14, 2012 at 05:20:13PM +0800, Alick Zhao wrote: >>> Yes the bug still occurs. >>> >>> I just wrote a simple script to ran the test for 50 times on T60 with >>> Ubuntu, GNU Radio 3.5.3 equipped, before and after FREQUENCY_OFFSET set >>> to 0. All failed the test. (n_pass is 0/50 in both sets) Every test's >>> output is almost identical in each set except particular test time >>> length. The ones before FREQUENCY_OFFSET change is basically the same as >>> test.t60.log already attached. One of the ones with FREQUENCY_OFFSET set >>> to 0 is attached below. I guess the almost same contents of output is >>> due to fixed random seed. >>> >>> I also ran the script on a Dell desktop with Fedora, and the result is >>> 50/50 pass the test. Then I notice one line in the qa python file says >>> that seed 1234 fails. However, 50/50 are OK with seed 1234 on the Dell >>> desktop. >> >> So, >> >> I tried this on a native 32-Bit Ubuntu 11.10, and it fails (tells me >> it's using "Volk machine: sse3_32". >> >> I also noticed the line that says seed=1234 fails. Also, the seed is set >> multiple times in the script. If this is the source of the bug or not, >> it should be fixed, because the seed is reset somewhere in the guts of >> the test. I'll post a patch on patch-gnuradio--this eliminates the >> problem on my machine, for some reason. If it does the same for you, >> this might actually be the solution. >> >> MB >> >> > > I applied the patch and tested it. No failure, yeah! > > However, with my debugging line, I can see this: > > [...] > constellation: <constellation psk (m=32)> differential: True correct: > 0.942307692308 > constellation: <constellation psk (m=64)> differential: True correct: > 0.772435897436 > constellation: <constellation psk (m=2)> differential: True correct: 1.0 > [...] > > So why 0.77XXX on the second line does not cause the assert to fail? > > alick > > _______________________________________________ > Discuss-gnuradio mailing list > Discuss-gnuradio@gnu.org > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
0001-Improving-qa_constellation_receiver.patch
Description: Binary data
_______________________________________________ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
