Bug reported: http://public.kitware.com/Bug/view.php?id=11877
On Fri, Feb 18, 2011 at 10:30 AM, David Cole <david.c...@kitware.com> wrote: > Yup. ctest -N and ctest without -N differ in test ordering. > > Congratulations! You've reported the first regression bug in CMake > 2.8.4....! :-) > > I suspect the change for this bug [ > http://public.kitware.com/Bug/view.php?id=11561 ] has introduced this > issue. I'll make a new issue that we can hopefully address fairly > soon. > > Thanks, > David > > > P.S. > ======= > The same build tree with a non -N ctest run gives me this: > > $ ctest > Test project /Users/davidcole/tmp/b6 > Start 27: enum > 1/52 Test #27: enum ............................. Passed 0.01 sec > Start 28: set_extent > 2/52 Test #28: set_extent ....................... Passed 3.66 sec > Start 29: getname > 3/52 Test #29: getname .......................... Passed 0.06 sec > Start 30: vfd > 4/52 Test #30: vfd .............................. Passed 0.03 sec > Start 31: ntypes > 5/52 Test #31: ntypes ........................... Passed 0.17 sec > Start 32: dangle > 6/52 Test #32: dangle ........................... Passed 0.18 sec > Start 33: dtransform > 7/52 Test #33: dtransform ....................... Passed 0.02 sec > Start 34: reserved > 8/52 Test #34: reserved ......................... Passed 0.00 sec > Start 35: cross_read > 9/52 Test #35: cross_read ....................... Passed 0.02 sec > Start 36: freespace > 10/52 Test #36: freespace ........................ Passed 0.03 sec > Start 37: mf > 11/52 Test #37: mf ............................... Passed 0.09 sec > ... > > > On Thu, Feb 17, 2011 at 10:41 PM, Allen D Byrne <b...@hdfgroup.org> wrote: >> David, >> >> I tried ctest -N ---- it lies! >> >> Here is the beginning of the test.log showing the problem (peviously the >> tests were in test number order) : >> >> Start testing: Feb 17 21:13 CST >> >> ---------------------------------------------------------- >> >> 435/660 Testing: H5COPY-clear-ext-links >> >> 436/660 Testing: H5COPY-ext_link >> >> 437/660 Testing: H5COPY-ext_link_f >> >> 438/660 Testing: H5COPY-ext_dangle_noobj >> >> 439/660 Testing: H5COPY-DIFF_ext_dangle_noobj >> >> 440/660 Testing: H5COPY-ext_dangle_noobj_f >> >> 441/660 Testing: H5COPY-ext_dangle_nofile >> >> 442/660 Testing: H5COPY-DIFF_ext_dangle_nofile >> >> 443/660 Testing: H5COPY-ext_dangle_nofile_f >> >> 444/660 Testing: H5COPY-ext_link_group >> >> 445/660 Testing: H5COPY-ext_link_group_f >> >> 446/660 Testing: H5COPY-H5LS_h5copy_extlinks_src >> >> 447/660 Testing: H5STAT-h5stat_help1 >> >> 448/660 Testing: H5STAT-h5stat_help2 >> >> 449/660 Testing: H5STAT-h5stat_filters >> >> 450/660 Testing: H5STAT-h5stat_filters-file >> >> 451/660 Testing: H5STAT-h5stat_filters-F >> >> 452/660 Testing: H5STAT-h5stat_filters-d >> >> 453/660 Testing: H5STAT-h5stat_filters-g >> >> 454/660 Testing: H5STAT-h5stat_filters-dT >> >> 414/660 Testing: H5COPY-DIFF_grp_nested >> >> 415/660 Testing: H5COPY-simple_group >> >> 416/660 Testing: H5COPY-DIFF_simple_group >> >> 417/660 Testing: H5COPY-grp_rename >> >> 418/660 Testing: H5COPY-DIFF_grp_rename >> >> 419/660 Testing: H5COPY-grp_dsets_rename >> >> 420/660 Testing: H5COPY-DIFF_grp_dsets_rename >> >> 421/660 Testing: H5COPY-A_B1_simple >> >> 422/660 Testing: H5COPY-DIFF_A_B1_simple >> >> 423/660 Testing: H5COPY-A_B2_simple2 >> >> 424/660 Testing: H5COPY-DIFF_A_B2_simple2 >> >> 425/660 Testing: H5COPY-C_D_simple >> >> 426/660 Testing: H5COPY-DIFF_C_D_simple >> >> 427/660 Testing: H5COPY-E_F_grp_dsets >> >> 428/660 Testing: H5COPY-DIFF_E_F_grp_dsets >> >> 429/660 Testing: H5COPY-G_H_grp_nested >> >> 430/660 Testing: H5COPY-DIFF_G_H_grp_nested >> >> 431/660 Testing: H5COPY-H5LS_h5copytst >> >> 432/660 Testing: H5COPY-clear-refs >> >> 433/660 Testing: H5COPY-region_ref >> >> 434/660 Testing: H5COPY-H5LS_h5copy_ref >> >> 476/660 Testing: H5DUMP-thlink-4 >> >> 477/660 Testing: H5DUMP-thlink-5 >> >> 478/660 Testing: H5DUMP-tcomp-1 >> >> 479/660 Testing: H5DUMP-tcomp-2 >> >> 480/660 Testing: H5DUMP-tcomp-4 >> >> 481/660 Testing: H5DUMP-tnestcomp-1 >> >> 482/660 Testing: H5DUMP-tall-1 >> >> 483/660 Testing: H5DUMP-tall-2 >> >> 484/660 Testing: H5DUMP-tall-3 >> >> 485/660 Testing: H5DUMP-tloop-1 >> >> 486/660 Testing: H5DUMP-tstr-1 >> >> 487/660 Testing: H5DUMP-tstr-2 >> >> 488/660 Testing: H5DUMP-tsaf >> >> 489/660 Testing: H5DUMP-tvldtypes1 >> >> 490/660 Testing: H5DUMP-tvldtypes2 >> >> 491/660 Testing: H5DUMP-tvldtypes3 >> >> 492/660 Testing: H5DUMP-tvldtypes4 >> >> 493/660 Testing: H5DUMP-tvldtypes5 >> >> 494/660 Testing: H5DUMP-tvlstr >> >> 495/660 Testing: H5DUMP-tarray1 >> >> 455/660 Testing: H5STAT-h5stat_filters-UD >> >> 456/660 Testing: H5STAT-h5stat_filters-UT >> >> 457/660 Testing: H5STAT-h5stat_tsohm >> >> 458/660 Testing: H5STAT-h5stat_newgrat >> >> 459/660 Testing: H5STAT-h5stat_newgrat-UG >> >> 460/660 Testing: H5STAT-h5stat_newgrat-UA >> >> 461/660 Testing: H5DUMP-tgroup-1 >> >> 462/660 Testing: H5DUMP-tgroup-2 >> >> 463/660 Testing: H5DUMP-tdset-1 >> >> 464/660 Testing: H5DUMP-tdset-2 >> >> 465/660 Testing: H5DUMP-tattr-1 >> >> 466/660 Testing: H5DUMP-tattr-2 >> >> 467/660 Testing: H5DUMP-tattr-3 >> >> 468/660 Testing: H5DUMP-tnamed_dtype_attr >> >> 469/660 Testing: H5DUMP-tslink-1 >> >> 470/660 Testing: H5DUMP-tudlink-1 >> >> 471/660 Testing: H5DUMP-tslink-2 >> >> 472/660 Testing: H5DUMP-tudlink-2 >> >> 473/660 Testing: H5DUMP-thlink-1 >> >> 474/660 Testing: H5DUMP-thlink-2 >> >> 475/660 Testing: H5DUMP-thlink-3 >> >> 352/660 Testing: H5REPACK-DIFF_chunk_compa >> >> 353/660 Testing: H5REPACK_VERIFY_LAYOUT-chunk_compa >> >> 354/660 Testing: H5REPACK-chunk_conti >> >> 355/660 Testing: H5REPACK-DIFF_chunk_conti >> >> 356/660 Testing: H5REPACK_VERIFY_LAYOUT-chunk_conti >> >> 357/660 Testing: H5REPACK-chunk_18x13 >> >> 358/660 Testing: H5REPACK-DIFF_chunk_18x13 >> >> 359/660 Testing: H5REPACK_VERIFY_LAYOUT-chunk_18x13 >> >> 360/660 Testing: H5REPACK-contig_small_compa >> >> 361/660 Testing: H5REPACK-DIFF_contig_small_compa >> >> 362/660 Testing: H5REPACK_VERIFY_LAYOUT-contig_small_compa >> >> 363/660 Testing: H5REPACK-contig_small_fixed_compa >> >> 364/660 Testing: H5REPACK-DIFF_contig_small_fixed_compa >> >> 365/660 Testing: H5REPACK_VERIFY_LAYOUT-contig_small_fixed_compa >> >> 366/660 Testing: H5REPACK-native_fill >> >> 367/660 Testing: H5REPACK-DIFF_native_fill >> >> 368/660 Testing: H5REPACK-native_attr >> >> 369/660 Testing: H5REPACK-DIFF_native_attr >> >> 370/660 Testing: H5REPACK-layout_long_switches >> >> 371/660 Testing: H5REPACK-DIFF_layout_long_switches >> >> 331/660 Testing: H5REPACK-DIFF_compa >> >> 332/660 Testing: H5REPACK_VERIFY_LAYOUT_ALL-compa >> >> 333/660 Testing: H5REPACK-dset_compa_conti >> >> 334/660 Testing: H5REPACK-DIFF_dset_compa_conti >> >> 335/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_compa_conti >> >> 336/660 Testing: H5REPACK-dset_compa_chunk >> >> 337/660 Testing: H5REPACK-DIFF_dset_compa_chunk >> >> 338/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_compa_chunk >> >> 339/660 Testing: H5REPACK-dset_compa_compa >> >> 340/660 Testing: H5REPACK-DIFF_dset_compa_compa >> >> 341/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_compa_compa >> >> 342/660 Testing: H5REPACK-dset_conti_compa >> >> 343/660 Testing: H5REPACK-DIFF_dset_conti_compa >> >> 344/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_conti_compa >> >> 345/660 Testing: H5REPACK-dset_conti_chunk >> >> 346/660 Testing: H5REPACK-DIFF_dset_conti_chunk >> >> 347/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_conti_chunk >> >> 348/660 Testing: H5REPACK-dset_conti_conti >> >> 349/660 Testing: H5REPACK-DIFF_dset_conti_conti >> >> 350/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_conti_conti >> >> 351/660 Testing: H5REPACK-chunk_compa >> >> 393/660 Testing: H5COPY-chunk >> >> 394/660 Testing: H5COPY-DIFF_chunk >> >> 395/660 Testing: H5COPY-compact >> >> 396/660 Testing: H5COPY-DIFF_compact >> >> 397/660 Testing: H5COPY-compound >> >> 398/660 Testing: H5COPY-DIFF_compound >> >> 399/660 Testing: H5COPY-compressed >> >> 400/660 Testing: H5COPY-DIFF_compressed >> >> 401/660 Testing: H5COPY-named_vl >> >> 402/660 Testing: H5COPY-DIFF_named_vl >> >> 403/660 Testing: H5COPY-nested_vl >> >> 404/660 Testing: H5COPY-DIFF_nested_vl >> >> 405/660 Testing: H5COPY-simple_top >> >> 406/660 Testing: H5COPY-DIFF_simple_top >> >> 407/660 Testing: H5COPY-rename >> >> 408/660 Testing: H5COPY-DIFF_rename >> >> 409/660 Testing: H5COPY-grp_empty >> >> 410/660 Test: H5COPY-DIFF_grp_empty >> >> 411/660 Testing: H5COPY-grp_dsets >> >> 412/660 Testing: H5COPY-DIFF_grp_dsets >> >> 413/660 Testing: H5COPY-grp_nested >> >> 372/660 Testing: H5REPACK_VERIFY_LAYOUT_ALL-layout_long_switches >> >> 373/660 Testing: H5REPACK-layout_short_switches >> >> 374/660 Test: H5REPACK-DIFF_layout_short_switches >> >> 375/660 Testing: H5REPACK_VERIFY_LAYOUT_ALL-layout_short_switches >> >> 376/660 Testing: H5REPACK-global_filters >> >> 377/660 Testing: H5REPACK-DIFF_global_filters >> >> 378/660 Testing: H5REPACK-add_userblock >> >> 379/660 Testing: H5REPACK-DIFF_add_userblock >> >> 380/660 Testing: H5REPACK-add_alignment >> >> 381/660 Testing: H5REPACK-DIFF_add_alignment >> >> 382/660 Testing: H5REPACK-upgrade_layout >> >> 383/660 Testing: H5REPACK-DIFF_upgrade_layout >> >> 384/660 Testing: H5REPACK-gt_mallocsize >> >> 385/660 Testing: H5REPACK-DIFF_gt_mallocsize >> >> 386/660 Testing: H5REPACK-committed_dt >> >> 387/660 Testing: H5REPACK-DIFF_committed_dt >> >> 388/660 Testing: H5REPACK-bug1814 >> >> 389/660 Testing: H5REPACK-DIFF_bug1814 >> >> 390/660 Testing: H5COPY-clear-objects >> >> 391/660 Testing: H5COPY-simple >> >> 392/660 Testing: H5COPY-DIFF_simple >> >> 600/660 Testing: H5DUMP-XML-thlink.h5 >> >> 601/660 Testing: H5DUMP-XML-tloop.h5 >> >> 602/660 Testing: H5DUMP-XML-tloop2.h5 >> >> 603/660 Testing: H5DUMP-XML-tmany.h5 >> >> 604/660 Testing: H5DUMP-XML-tnestedcomp.h5 >> >> 605/660 Testing: H5DUMP-XML-tcompound_complex.h5 >> >> Allen >> >>> After building HDF5 v 1.8.6 on my Mac, I get this: >> >>> >> >>> $ ctest -N >> >>> Test project /Users/davidcole/tmp/b6 >> >>> Test #1: testhdf5 >> >>> Test #2: accum >> >>> Test #3: lheap >> >>> Test #4: ohdr >> >>> Test #5: stab >> >>> Test #6: gheap >> >>> Test #7: pool >> >>> Test #8: hyperslab >> >>> Test #9: istore >> >>> Test #10: bittests >> >>> Test #11: dt_arith >> >>> Test #12: dtypes >> >>> Test #13: dsets >> >>> Test #14: cmpd_dset >> >>> Test #15: extend >> >>> Test #16: external >> >>> Test #17: objcopy >> >>> Test #18: links >> >>> Test #19: unlink >> >>> Test #20: big >> >>> Test #21: mtime >> >>> Test #22: fillval >> >>> Test #23: mount >> >>> Test #24: flush1 >> >>> Test #25: flush2 >> >>> Test #26: app_ref >> >>> Test #27: enum >> >>> Test #28: set_extent >> >>> Test #29: getname >> >>> Test #30: vfd >> >>> Test #31: ntypes >> >>> Test #32: dangle >> >>> Test #33: dtransform >> >>> Test #34: reserved >> >>> Test #35: cross_read >> >>> Test #36: freespace >> >>> Test #37: mf >> >>> Test #38: btree2 >> >>> Test #39: fheap >> >>> Test #40: error_test >> >>> Test #41: err_compat >> >>> Test #42: tcheck_version >> >>> Test #43: testmeta >> >>> Test #44: cache >> >>> Test #45: cache_api >> >>> Test #46: ttsafe >> >>> Test #47: h5perf_serial >> >>> Test #48: chunk >> >>> Test #49: iopipe >> >>> Test #50: overhead >> >>> Test #51: perf_meta >> >>> Test #52: zip_perf >> >>> >> >>> Total Tests: 52 >> >>> >> >>> I'm not familiar enough with HDF5 to know if this is correct or not. >> >>> Is this the same order you're seeing? >> >>> >> >>> Or can you tell me your expected order? I'll try to analyze the source >> >>> tree and figure out what's going on, but what you expect will tell me >> >>> something, too. >> >>> >> >>> >> >>> Thanks, >> >>> David >> >>> >> >>> >> >>> >> >>> On Thu, Feb 17, 2011 at 5:10 PM, Allen D Byrne <b...@hdfgroup.org> wrote: >> >>> > Just using standard gcc/gfortran on the fedora distro. It is the 64bit >> >>> > version? >> >>> > >> >>> > You can grab one of our recent src tarballs from >> >>> > >> >>> > "http://www.hdfgroup.org/ftp/HDF5/hdf5-1.8.6/" >> >>> > >> >>> > Allen >> >>> > >> >>> > PS. I will be out of touch for the next 36 hours >> >>> > >> >>> >> On Thu, Feb 17, 2011 at 4:20 PM, Allen D Byrne <b...@hdfgroup.org> >>> >> wrote: >> >>> > >> >>> >> > This was from a clean build folder. And I just want the test to run >>> >> > in >> >>> >> > the >> >>> > >> >>> >> > order defined in the CMakeLists.txt files. Also never had set any >>> >> > COST >> >>> > >> >>> >> > properties. The surprise was that this never happened before. >> >>> > >> >>> >> > >> >>> > >> >>> >> > Allen >> >>> > >> >>> >> > >> >>> > >> >>> >> >> On Thu, Feb 17, 2011 at 4:08 PM, David Cole <david.c...@kitware.com> >> >>> > >> >>> >> >> wrote: >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > On Thu, Feb 17, 2011 at 3:57 PM, Allen D Byrne <b...@hdfgroup.org> >> >>> > >> >>> >> >> > wrote: >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> I just installed 2.8.4 and on my fedora 14 linux box the ctest -D >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> Experimental stage decided to run the tests in some unknown to me >> >>> > >> >>> >> >> >> order. The >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> previous versions always ran them in order of definition. Any >>> >> >> >> thing >> >>> >> >> >> I >> >>> > >> >>> >> >> >> can do >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> to get the ordering back? >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> Allen >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> _______________________________________________ >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> Powered by www.kitware.com >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> Visit other Kitware open-source projects at >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> http://www.kitware.com/opensource/opensource.html >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> Please keep messages on-topic and check the CMake FAQ at: >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> http://www.cmake.org/Wiki/CMake_FAQ >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> Follow this link to subscribe/unsubscribe: >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> http://www.cmake.org/mailman/listinfo/cmake >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > Delete the file "Testing/Temporary/CTestCostData.txt" in your >>> >> >> > build >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > tree. It saves run times from run to run and orders them as >>> >> >> > "slowest >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > first" on subsequent runs. >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > A clean build (no CostData file) without any parallel testing, and >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > without any test COST properties defined should give you the top >>> >> >> > to >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > bottom ordering that you're used to. >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > Saving this data from run to run helps us schedule parallel >>> >> >> > testing >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > jobs efficiently on subsequent runs. >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > HTH, >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > David >> >>> > >> >>> >> > >> >>> > >> >>> >> >> > >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> >>> > >> >>> >> > >> >>> > >> >>> >> >> Having said all that: if your goal is to order certain tests with >> >>> > >> >>> >> > >> >>> > >> >>> >> >> respect to each other, see the help for the test property "DEPENDS": >> >>> > >> >>> >> > >> >>> > >> >>> >> >> http://cmake.org/cmake/help/cmake-2-8-docs.html#prop_test:DEPENDS >> >>> > >> >>> >> > >> >>> > >> >>> >> >> >> >>> > >> >>> >> >> >>> > >> >>> >> So there is no "Testing/Temporary/CTestCostData.txt" file at the time >> >>> > >> >>> >> you call "ctest -D Experimental"? >> >>> > >> >>> >> >> >>> > >> >>> >> Can you point me to a source tree that I can build that demonstrates >> >>> > >> >>> >> this behavior? >> >>> > >> >>> >> >> >>> > >> >>> >> If you are not running tests in parallel and you have no "cost data" >> >>> > >> >>> >> (in the form of a cached file or properties set) then the tests should >> >>> > >> >>> >> run in the same top-to-bottom order that they always have. >> >>> > >> >>> >> >> >>> > >> >>> >> >> >>> > >> >>> >> Thx, >> >>> > >> >>> >> David >> >>> > >> >>> >> >> >>> > _______________________________________________ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake