Re: [CMake] Code Coverage Bullseye
Thou shall receive the patch :) On Thu, Sep 18, 2014 at 5:56 PM, David Cole dlrd...@aol.com wrote: My apologies. I take it back -- gcov apparently does work on the Mac nowadays. It is a ctest issue, then, and when using Xcode, ctest will need to look in some additional locations for the coverage data files. The source code is available -- patches welcome. :-) D -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] Code Coverage Bullseye
Hi, Thanks for your responses. As David mentioned I don't have to set the COVFILE if I don't have the Bullseye. I am already using the -fprofile-arcs -ftest-coverage hence .gcna .gcno files are getting generated. Here is my CMakeLists.txt : http://pastebin.com/hQt2htkK Here is my ctest script : http://pastebin.com/CFqHkPNZ I also enabled the --Debug flag for ctest script and I found out that ctest is looking the gcov files in following directories: 1. Performing coverage 2. COVFILE environment variable not found, not running bullseye 3.globbing for coverage in: /Users/rajekumar/work/test-codes/Test/out/CMakeFiles/ALL_BUILD.dir 4.globbing for coverage in: /Users/rajekumar/work/test-codes/Test/out/CMakeFiles/Continuous.dir 5.globbing for coverage in: /Users/rajekumar/work/test-codes/Test/out/CMakeFiles/Experimental.dir 6.globbing for coverage in: /Users/rajekumar/work/test-codes/Test/out/CMakeFiles/Nightly.dir 7.globbing for coverage in: /Users/rajekumar/work/test-codes/Test/out/CMakeFiles/NightlyMemoryCheck.dir 8.globbing for coverage in: /Users/rajekumar/work/test-codes/Test/out/CMakeFiles/ZERO_CHECK.dir 9.globbing for coverage in: /Users/rajekumar/work/test-codes/Test/out/intercepttest1/CMakeFiles/intercepttest1.dir 10.globbing for coverage in: /Users/rajekumar/work/test-codes/Test/out/intercepttest2/CMakeFiles/intercepttest2.dir 11.globbing for coverage in: /Users/rajekumar/work/test-codes/Test/out/sporttest1/CMakeFiles/sporttest1.dir 12.globbing for coverage in: /Users/rajekumar/work/test-codes/Test/out/sporttest2/CMakeFiles/sporttest2.dir 13.globbing for coverage in: /Users/rajekumar/work/test-codes/Test/out/test/CMakeFiles/Sample.dir 14. Cannot find any GCov coverage files. 15. Cannot find any Python Trace.py coverage files. 16. Cannot find coverage.py XML file: /Users/rajekumar/work/test-codes/Test/out/coverage.xml 17. Cannot find foobar GTM coverage file: /Users/rajekumar/work/test-codes/Test/out/gtm_coverage.mcov 18. Cannot find Cache coverage file: /Users/rajekumar/work/test-codes/Test/out/cache_coverage.cmcov 19. Cannot find any coverage files. Ignoring Coverage request. But simple grep shows that .gcda .gcno are located in different directory: gcno files: ./out/intercepttest1/Project.build/Debug/intercepttest1.build/Objects-normal/x86_64/intercepttest1.gcno ./out/intercepttest2/Project.build/Debug/intercepttest2.build/Objects-normal/x86_64/intercepttest2.gcno ./out/sporttest1/Project.build/Debug/sporttest1.build/Objects-normal/x86_64/sporttest1.gcno ./out/sporttest2/Project.build/Debug/sporttest2.build/Objects-normal/x86_64/sporttest2.gcno ./out/test/Project.build/Debug/Sample.build/Objects-normal/x86_64/main.gcno gcda files: ./out/intercepttest1/Project.build/Debug/intercepttest1.build/Objects-normal/x86_64/intercepttest1.gcda ./out/intercepttest2/Project.build/Debug/intercepttest2.build/Objects-normal/x86_64/intercepttest2.gcda ./out/sporttest1/Project.build/Debug/sporttest1.build/Objects-normal/x86_64/sporttest1.gcda ./out/sporttest2/Project.build/Debug/sporttest2.build/Objects-normal/x86_64/sporttest2.gcda ./out/test/Project.build/Debug/Sample.build/Objects-normal/x86_64/main.gcda Is this the problem ??? Here is the full debug logs : http://pastebin.com/cTXFZ3tS Thanks Rajeev On Wed, Sep 17, 2014 at 6:26 PM, Bill Hoffman bill.hoff...@kitware.com wrote: On 9/17/2014 8:47 AM, NoRulez wrote: Hello, since code coverage doesn't work very well with gcov and so on on Mac, our company decided to use bullseye for Mac OS too. If you could email the COVFILE to me I could take a look. Also it might help if you could run ctest with --debug and -VV and send those results to me as well. Sounds like and issue with ctest. -Bill -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/ opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source
Re: [CMake] Code Coverage Bullseye
As I said in my previous response, I don't know anybody who is using Xcode on the Mac, and also successfully producing a gcov coverage build. If I were you, I would: (1) use gcov on a Linux build of your software for coverage measurement, using the Unix Makefiles generator and the right gcov flags or (2) buy Bullseye for the Mac, as NoRulez suggested his company did because they couldn't get a reliable coverage build using gcov on the Mac This is not a ctest problem. This is a gcov doesn't work right on the Mac problem. If somebody would like to claim gcov does work right on the Mac, I would love to see the proof. If anybody is doing it successfully, please do share your scripts and a link to the dashboard results you get using it. (And any hints we might need to replicate it ourselves...) Thanks, David C. -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] Code Coverage Bullseye
As David suggested I did following experiment. I created two Test projects. *Project1: * *set(CTEST_SOURCE_DIRECTORY /Users/rajekumar/work/test-codes/TestUnix)* *set(CTEST_BINARY_DIRECTORY /Users/rajekumar/work/test-codes/TestUnix/out)* *set(CTEST_CMAKE_GENERATOR Unix Makefiles)* *set(CTEST_BUILD_COMMAND make)* *Project2:* *set(CTEST_SOURCE_DIRECTORY /Users/rajekumar/work/test-codes/TestMac)* *set(CTEST_BINARY_DIRECTORY /Users/rajekumar/work/test-codes/TestMac/out)* *set(CTEST_CMAKE_GENERATOR Xcode)* *set(CTEST_BUILD_COMMAND xcodebuild)* Project1 is able to do the code coverage.Here is the log snippet. *SetCTestConfiguration:CoverageCommand:/usr/bin/gcov* * Add coverage exclude regular expressions.* * target directory list [/Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/TargetDirectories.txt]* *Performing coverage* * COVFILE environment variable not found, not running bullseye* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/Continuous.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ContinuousBuild.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ContinuousConfigure.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ContinuousCoverage.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ContinuousMemCheck.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ContinuousStart.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ContinuousSubmit.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ContinuousTest.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ContinuousUpdate.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/Experimental.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ExperimentalBuild.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ExperimentalConfigure.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ExperimentalCoverage.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ExperimentalMemCheck.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ExperimentalStart.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ExperimentalSubmit.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ExperimentalTest.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/ExperimentalUpdate.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/Nightly.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/NightlyBuild.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/NightlyConfigure.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/NightlyCoverage.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/NightlyMemCheck.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/NightlyMemoryCheck.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/NightlyStart.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/NightlySubmit.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/NightlyTest.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/CMakeFiles/NightlyUpdate.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestUnix/out/test/CMakeFiles/Sample.dir* * Processing coverage (each . represents one file):* *./usr/bin/gcov -o /Users/rajekumar/work/test-codes/TestUnix/out/test/CMakeFiles/Sample.dir /Users/rajekumar/work/test-codes/TestUnix/out/test/CMakeFiles/Sample.dir/main.c.gcda* *File '/Users/rajekumar/work/test-codes/TestUnix/test/main.c'* *Lines executed:100.00% of 2* In Project 2 ctest was unable to find the .gcda. It was in /Users/rajekumar/work/test-codes/TestMac/out/test/Project.build/Debug/Sample.build/Objects-normal/x86_64/ and ctest was searching in following directories(from logs): *globbing for coverage in: /Users/rajekumar/work/test-codes/TestMac/out/CMakeFiles/ALL_BUILD.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestMac/out/CMakeFiles/Continuous.dir* * globbing for coverage in: /Users/rajekumar/work/test-codes/TestMac/out/CMakeFiles/Experimental.dir* * globbing for coverage in:
Re: [CMake] Code Coverage Bullseye
My apologies. I take it back -- gcov apparently does work on the Mac nowadays. It is a ctest issue, then, and when using Xcode, ctest will need to look in some additional locations for the coverage data files. The source code is available -- patches welcome. :-) D -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] Code Coverage Bullseye
I am trying to configure the code coverage using the ctest script. I took help from : http://www.cmake.org/Wiki/CTest/Coverage That wiki page was mostly written in 2007 and 2009. You may want to refer to something more recent than that, and something that's proven to work, like an existing dashboard script that does coverage, as a reference. One thing to note: there may not be a working example of somebody doing code coverage using the Xcode generator on a Mac. If there is, hopefully he will speak up here, and give you some additional hints. You might have better luck doing a coverage dashboard on Linux using gcc and gcov. See the scripts used for coverage dashboards on the CMake dashboard... for example, this one: http://open.cdash.org/viewNotes.php?buildid=3493343 Especially note the use of the -fprofile-arcs -ftest-coverage flags in the script. (Mentioned on that wiki page, I might add.) Is Bullseye is mandatory for code coverage using ctest ??? Is there any way to perform code coverage through ctest without using bullseye ?? Bullseye is absolutely NOT mandatory for code coverage testing using ctest, it's simply one of the available options. If you do not have Bullseye, there is no need to set the COVFILE env variable. HTH, David C. -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] Code Coverage Bullseye
Hello, since code coverage doesn't work very well with gcov and so on on Mac, our company decided to use bullseye for Mac OS too. I hope it helps Am 17.09.2014 um 14:22 schrieb David Cole via CMake cmake@cmake.org: I am trying to configure the code coverage using the ctest script. I took help from : http://www.cmake.org/Wiki/CTest/Coverage That wiki page was mostly written in 2007 and 2009. You may want to refer to something more recent than that, and something that's proven to work, like an existing dashboard script that does coverage, as a reference. One thing to note: there may not be a working example of somebody doing code coverage using the Xcode generator on a Mac. If there is, hopefully he will speak up here, and give you some additional hints. You might have better luck doing a coverage dashboard on Linux using gcc and gcov. See the scripts used for coverage dashboards on the CMake dashboard... for example, this one: http://open.cdash.org/viewNotes.php?buildid=3493343 Especially note the use of the -fprofile-arcs -ftest-coverage flags in the script. (Mentioned on that wiki page, I might add.) Is Bullseye is mandatory for code coverage using ctest ??? Is there any way to perform code coverage through ctest without using bullseye ?? Bullseye is absolutely NOT mandatory for code coverage testing using ctest, it's simply one of the available options. If you do not have Bullseye, there is no need to set the COVFILE env variable. HTH, David C. -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] Code Coverage Bullseye
On 9/17/2014 8:47 AM, NoRulez wrote: Hello, since code coverage doesn't work very well with gcov and so on on Mac, our company decided to use bullseye for Mac OS too. If you could email the COVFILE to me I could take a look. Also it might help if you could run ctest with --debug and -VV and send those results to me as well. Sounds like and issue with ctest. -Bill -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
[CMake] Code Coverage Bullseye
Hi,
I am trying to configure the code coverage using the ctest script. I took
help from : http://www.cmake.org/Wiki/CTest/Coverage
My script looks like this:
*cmake_minimum_required(VERSION 3.0)set(CTEST_SOURCE_DIRECTORY
/Users/rajekumar/work/test-codes/Test)set(CTEST_BINARY_DIRECTORY
/Users/rajekumar/work/test-codes/Test/out/)#set(CTEST_SITE
http://oak-vcs187.lab.nbttech.com
http://oak-vcs187.lab.nbttech.com)set(CTEST_BUILD_NAME
MacOSX)set(CTEST_CMAKE_GENERATOR Xcode)set(CTEST_BUILD_COMMAND
xcodebuild)set(CTEST_COVERAGE_COMMAND /usr/bin/gcov)set(ENV{COVFILE}
${CTEST_BINARY_DIRECTORY}/CMake.cov)ctest_empty_binary_directory(${CTEST_BINARY_DIRECTORY})ctest_start(Experimental)#ctest_update()set(CTEST_PROJECT_SUBPROJECTS
SportTest SampleTest InterceptTest)foreach(subproject
${CTEST_PROJECT_SUBPROJECTS})set_property(GLOBAL PROPERTY SubProject
${subproject})set_property(GLOBAL PROPERTY Label ${subproject})
ctest_configure()set(CTEST_BUILD_TARGET ${subproject})
ctest_build(APPEND)ctest_test(INCLUDE_LABEL ${subproject})
ctest_coverage()ctest_submit()endforeach()*
*But after running I am getting following error:Performing coverage run
covsrc with
COVFILE=[/Users/rajekumar/work/test-codes/Test/out//CMake.cov]Cannot find
:covsrcerror running covsrc:Error running bullseye summary. globbing for
coverage in:
/Users/rajekumar/work/test-codes/Test/out/CMakeFiles/ALL_BUILD.dir
globbing for coverage in:
/Users/rajekumar/work/test-codes/Test/out/CMakeFiles/Continuous.dir
globbing for coverage in:
/Users/rajekumar/work/test-codes/Test/out/CMakeFiles/Experimental.dir
globbing for coverage in:
/Users/rajekumar/work/test-codes/Test/out/CMakeFiles/Nightly.dir globbing
for coverage in:
/Users/rajekumar/work/test-codes/Test/out/CMakeFiles/NightlyMemoryCheck.dir
globbing for coverage in:
/Users/rajekumar/work/test-codes/Test/out/CMakeFiles/ZERO_CHECK.dir
globbing for coverage in:
/Users/rajekumar/work/test-codes/Test/out/intercepttest1/CMakeFiles/intercepttest1.dir
globbing for coverage in:
/Users/rajekumar/work/test-codes/Test/out/intercepttest2/CMakeFiles/intercepttest2.dir
globbing for coverage in:
/Users/rajekumar/work/test-codes/Test/out/sporttest1/CMakeFiles/sporttest1.dir
globbing for coverage in:
/Users/rajekumar/work/test-codes/Test/out/sporttest2/CMakeFiles/sporttest2.dir
globbing for coverage in:
/Users/rajekumar/work/test-codes/Test/out/test/CMakeFiles/Sample.dir Cannot
find any GCov coverage files. Cannot find any Python Trace.py coverage
files.Cannot find coverage.py XML file:
/Users/rajekumar/work/test-codes/Test/out/coverage.xml Cannot find foobar
GTM coverage file:
/Users/rajekumar/work/test-codes/Test/out/gtm_coverage.mcov Cannot find
Cache coverage file:
/Users/rajekumar/work/test-codes/Test/out/cache_coverage.cmcov Cannot find
any coverage files. Ignoring Coverage request.*
I was under the impression that I only need gcov lcov for code coverage.
Is Bullseye is mandatory for code coverage using ctest ???
Is there any way to perform code coverage through ctest without using
bullseye ??
Thanks.
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
