Hi all,
I am using CMake again since a couple of weeks for a project involving the
build of recognition system. This system is made of hundred sub recognition
system that must be all trained with about ~10k files as input.
I ended up writing something like that:
====== BEGIN
# A project with hundred targets depending on one command with 10k
dependencies.
cmake_minimum_required(VERSION 2.8)
project(HundredTargetsDepOnOneCommandWithManyDeps)
include(CTest)
# Define LARGE_LIST with 10000 input files.
include(large_list.cmake)
foreach(i ${LARGE_LIST})
file(WRITE ${i} "input")
endforeach(i)
add_custom_command(
OUTPUT bottle_neck
COMMAND ${CMAKE_COMMAND} -E touch bottle_neck
DEPENDS ${LARGE_LIST}
COMMENT "Executing command with large number of dependencies..."
)
set(OUTPUTS )
foreach(i RANGE 100)
set(_output output_${i})
add_custom_target(${_output}-target ALL
DEPENDS bottle_neck
)
list(APPEND OUTPUTS ${_output})
endforeach(i)
add_executable(AlwaysTrue AlwaysTrue.c)
====== END
Note that the hundred targets have only one dependency. Running CMake on
this project is slow. The configuration is fast but the generation is very
slow. Here some examples:
With Unix Makefiles
$ time make rebuild_cache
Running CMake to regenerate build system...
-- Configuration done
(this is long)
-- Generation done
(this is long)
-- Build files have been written to: ...
make rebuild_cache 120.54s user 6.08s system 97% cpu 2:10.07 total
Memory used is normal (htop indicates 82488 VIRT) and CPU is at 100%.
$ time make
make 129.46s user 3.76s system 99% cpu 2:14.28 total
With Ninja:
$ time ninja -v rebuild_cache
ninja rebuild_cache 98.89s user 0.10s system 99% cpu 1:39.02 total
$ time ninja
ninja 0.01s user 0.04s system 86% cpu 0.055 total
So here my two questions:
1/ Do you think it is possible to improve the performance? (I have not
profile it yet).
2/ Is there a way to set a timeout to ctest so that my test fail. I have
that in Tests/CMakeLists.txt but it does not work:
set(_test_name HundredTargetsDepOnOneCommandWithManyDeps)
add_test(${_test_name} ${CMAKE_CTEST_COMMAND}
--build-and-test
"${CMake_SOURCE_DIR}/Tests/${_test_name}"
"${CMake_BINARY_DIR}/Tests/${_test_name}"
--build-two-config
${build_generator_args}
--build-project ${_test_name}
--timeout 10
--test-command AlwaysTrue)
list(APPEND TEST_BUILD_DIRS
"${CMake_BINARY_DIR}/Tests/${_test_name}")
unset(_test_name)
I have pushed my work so far on my github clone of CMake.
https://github.com/nicolasdespres/CMake/tree/topic/large-deps-perf
I am willing to do the work on this topic but I wanted to have your insight
first.
Thanks,
--
Nicolas Desprès
--
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
