Re: [CMake] CMake Project Generation Speedup

[Robert Maynard via CMake]

A round of performance improvements to generate time was done as part
of CMake 3.11 and that significantly helped. What would be helpful is
a performance analysis run of CMake itself to determine if the issue
is that we are IO bound ( and need to do multi-threaded writes ) or
compute bound.

While this information will be project dependent, it would be great to
start getting samples from the community.

On Wed, Mar 20, 2019 at 1:53 PM J. Caleb Wherry  wrote:
>
> Did anything ever come of this?
>
> I am in a similar boat: we have >800 targets on our full build (native C++, 
> Managed C++, C#, Java, CUDA, etc) and the majority of the time for the 
> configure/generate steps takes place in the generate step (>70%).
>
> I understand there is a lot of IO since the generate step has to write the 
> project files and filters for each C++ project (the majority of our projects) 
> for VS generators (what we use). I'm just looking to see if there is anything 
> to look at or potentially speedup up the generate step besides "get a faster 
> drive".
>
> Thanks!
> -Caleb
>
> On Thu, Nov 17, 2016 at 11:15 AM Damian  wrote:
>>
>> Hi all,
>>
>> We are still in the process of switching our large Make-based build to 
>> CMake. One of the issues we're running into is the time it takes to reparse 
>> and regenerate the CMake project (whether ninja, VS, or make) after touching 
>> any CMake file. To give you an idea, we have about 1000 targets and that 
>> takes a good 2 min for CMake to rerun.
>>
>> Are there any plans to speed this up? Maybe parallelize it in some way or do 
>> a better job regenerating only what needs regenerating? Is there anything we 
>> can do on our side to reduce our regeneration times?
>>
>> For example, if using a VS generator, each directory in the source that has 
>> a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch one 
>> of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.
>>
>> Thanks for any help.
>> --
>>
>> Powered by www.kitware.com
>>
>> Please keep messages on-topic and check the CMake FAQ at: 
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Kitware offers various services to support the CMake community. For more 
>> information on each offering, please visit:
>>
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>
>> Visit other Kitware open-source projects at 
>> http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake
>
>
>
> --
> J. Caleb Wherry
> Scientific Software Engineer
>
> http://www.calebwherry.com
> +1 (615) 708-5651
> calebwhe...@gmail.com
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more 
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

Re: [CMake] CMake Project Generation Speedup

[Cristian Adam]

Hi,

CMake has for Visual Studio is a multi configuration generator. (Debug,
Release, RelMinSize, RelWithDbgInformation).

If you specify only one configuration you should cut the generation time to
1/4th.

See
https://cmake.org/cmake/help/latest/variable/CMAKE_CONFIGURATION_TYPES.html#variable:CMAKE_CONFIGURATION_TYPES

Cheers,
Cristian

On Wed, Mar 20, 2019, 21:53 J. Caleb Wherry  wrote:

> Did anything ever come of this?
>
> I am in a similar boat: we have >800 targets on our full build (native
> C++, Managed C++, C#, Java, CUDA, etc) and the majority of the time for the
> configure/generate steps takes place in the generate step (>70%).
>
> I understand there is a lot of IO since the generate step has to write the
> project files and filters for each C++ project (the majority of our
> projects) for VS generators (what we use). I'm just looking to see if there
> is anything to look at or potentially speedup up the generate step besides
> "get a faster drive".
>
> Thanks!
> -Caleb
>
> On Thu, Nov 17, 2016 at 11:15 AM Damian  wrote:
>
>> Hi all,
>>
>> We are still in the process of switching our large Make-based build to
>> CMake. One of the issues we're running into is the time it takes to reparse
>> and regenerate the CMake project (whether ninja, VS, or make) after
>> touching any CMake file. To give you an idea, we have about 1000 targets
>> and that takes a good 2 min for CMake to rerun.
>>
>> Are there any plans to speed this up? Maybe parallelize it in some way or
>> do a better job regenerating only what needs regenerating? Is there
>> anything we can do on our side to reduce our regeneration times?
>>
>> For example, if using a VS generator, each directory in the source that
>> has a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch
>> one of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.
>>
>> Thanks for any help.
>> --
>>
>> Powered by www.kitware.com
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Kitware offers various services to support the CMake community. For more
>> information on each offering, please visit:
>>
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake
>
>
>
> --
> J. Caleb Wherry
> *Scientific Software Engineer*
>
> 
> http://www.calebwherry.com
> +1 (615) 708-5651
> calebwhe...@gmail.com
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

Re: [CMake] CMake Project Generation Speedup

[Gregor Jasny via CMake]

Hello,

On 17.11.16 17:15, Damian wrote:
We are still in the process of switching our large Make-based build to 
CMake. One of the issues we're running into is the time it takes to 
reparse and regenerate the CMake project (whether ninja, VS, or make) 
after touching any CMake file. To give you an idea, we have about 1000 
targets and that takes a good 2 min for CMake to rerun.


Are there any plans to speed this up?


CMake VisualStudio and Xcode generator generates one solution / 
xcodeproj per project() commmand. Due to the shared vcxproj files this 
is not too expensive for Visual Studio. But for Xcode it is. You can 
disable that behavior by specifying


  -DCMAKE_XCODE_GENERATE_TOP_LEVEL_PROJECT_ONLY=ON

on the command line.

Thanks,
-Gregor
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

Re: [CMake] CMake Project Generation Speedup

[Kuba Ober]

The best advice I had for myself: profile the code and tweak it. The last I 
looked, and that was a while ago, the generators could use a threaded write 
queue so that the file output would overlap with file generation. Those 
probably could be memory mapped files, too, to further minimize the overhead: 
writing to mapped pages that are dumped to disk via DMA (that’s how page 
flushes work) is cheaper than copying all the data in little chunks into a 
small buffer in the file output code, and then once again into the page that 
ends up flushed to disk. Never mind that the write latencies are on the hot 
path, where they don’t belong.

If generators could be const-correct (not writing anywhere into the Makefile 
data structures), they could all be parallelized as well – but I’m not sure 
whether that’s a minor hack or a major rework. 

Cheers, Kuba

> 20 mars 2019 kl. 16:53 skrev J. Caleb Wherry :
> 
> Did anything ever come of this?
> 
> I am in a similar boat: we have >800 targets on our full build (native C++, 
> Managed C++, C#, Java, CUDA, etc) and the majority of the time for the 
> configure/generate steps takes place in the generate step (>70%).
> 
> I understand there is a lot of IO since the generate step has to write the 
> project files and filters for each C++ project (the majority of our projects) 
> for VS generators (what we use). I'm just looking to see if there is anything 
> to look at or potentially speedup up the generate step besides "get a faster 
> drive".
> 
> Thanks!
> -Caleb
> 
>> On Thu, Nov 17, 2016 at 11:15 AM Damian  wrote:
>> Hi all,
>> 
>> We are still in the process of switching our large Make-based build to 
>> CMake. One of the issues we're running into is the time it takes to reparse 
>> and regenerate the CMake  project (whether ninja, VS, or make) after 
>> touching any CMake file. To give you an idea, we have about 1000 targets and 
>> that takes a good 2 min for CMake to rerun.
>> 
>> Are there any plans to speed this up? Maybe parallelize it in some way or do 
>> a better job regenerating only what needs regenerating? Is there anything we 
>> can do on our side to reduce our regeneration times?
>> 
>> For example, if using a VS generator, each directory in the source that has 
>> a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch one 
>> of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.
>> 
>> Thanks for any help.
>> -- 
>> 
>> Powered by www.kitware.com
>> 
>> Please keep messages on-topic and check the CMake FAQ at: 
>> http://www.cmake.org/Wiki/CMake_FAQ
>> 
>> Kitware offers various services to support the CMake community. For more 
>> information on each offering, please visit:
>> 
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>> 
>> Visit other Kitware open-source projects at 
>> http://www.kitware.com/opensource/opensource.html
>> 
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake
> 
> 
> -- 
> J. Caleb Wherry
> Scientific Software Engineer
> 
> http://www.calebwherry.com
> +1 (615) 708-5651
> calebwhe...@gmail.com
> -- 
> 
> Powered by www.kitware.com
> 
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
> 
> Kitware offers various services to support the CMake community. For more 
> information on each offering, please visit:
> 
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
> 
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
> 
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

Re: [CMake] CMake Project Generation Speedup

[Kyle Edwards via CMake]

On Wed, 2019-03-20 at 16:57 -0400, J. Caleb Wherry wrote:
> I was also surprised when "cmake --trace" gave 0 information related
> to the generate step. I assume this is expected behavior?
The purpose of --trace is to debug CMake scripts. No scripts get run
during the generate step, so yes, this is expected.
Kyle-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

Re: [CMake] CMake Project Generation Speedup

[J. Caleb Wherry]

I was also surprised when "cmake --trace" gave 0 information related to the
generate step. I assume this is expected behavior?

Oh and:

CMake: 3.13.4
Visual Studio 2017 15.9.9
Win7

-Caleb

On Wed, Mar 20, 2019 at 4:53 PM J. Caleb Wherry 
wrote:

> Did anything ever come of this?
>
> I am in a similar boat: we have >800 targets on our full build (native
> C++, Managed C++, C#, Java, CUDA, etc) and the majority of the time for the
> configure/generate steps takes place in the generate step (>70%).
>
> I understand there is a lot of IO since the generate step has to write the
> project files and filters for each C++ project (the majority of our
> projects) for VS generators (what we use). I'm just looking to see if there
> is anything to look at or potentially speedup up the generate step besides
> "get a faster drive".
>
> Thanks!
> -Caleb
>
> On Thu, Nov 17, 2016 at 11:15 AM Damian  wrote:
>
>> Hi all,
>>
>> We are still in the process of switching our large Make-based build to
>> CMake. One of the issues we're running into is the time it takes to reparse
>> and regenerate the CMake project (whether ninja, VS, or make) after
>> touching any CMake file. To give you an idea, we have about 1000 targets
>> and that takes a good 2 min for CMake to rerun.
>>
>> Are there any plans to speed this up? Maybe parallelize it in some way or
>> do a better job regenerating only what needs regenerating? Is there
>> anything we can do on our side to reduce our regeneration times?
>>
>> For example, if using a VS generator, each directory in the source that
>> has a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch
>> one of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.
>>
>> Thanks for any help.
>> --
>>
>> Powered by www.kitware.com
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Kitware offers various services to support the CMake community. For more
>> information on each offering, please visit:
>>
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake
>
>
>
> --
> J. Caleb Wherry
> *Scientific Software Engineer*
>
> 
> http://www.calebwherry.com
> +1 (615) 708-5651
> calebwhe...@gmail.com
>


-- 
J. Caleb Wherry
*Scientific Software Engineer*


http://www.calebwherry.com
+1 (615) 708-5651
calebwhe...@gmail.com
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

Re: [CMake] CMake Project Generation Speedup

[J. Caleb Wherry]

Did anything ever come of this?

I am in a similar boat: we have >800 targets on our full build (native C++,
Managed C++, C#, Java, CUDA, etc) and the majority of the time for the
configure/generate steps takes place in the generate step (>70%).

I understand there is a lot of IO since the generate step has to write the
project files and filters for each C++ project (the majority of our
projects) for VS generators (what we use). I'm just looking to see if there
is anything to look at or potentially speedup up the generate step besides
"get a faster drive".

Thanks!
-Caleb

On Thu, Nov 17, 2016 at 11:15 AM Damian  wrote:

> Hi all,
>
> We are still in the process of switching our large Make-based build to
> CMake. One of the issues we're running into is the time it takes to reparse
> and regenerate the CMake project (whether ninja, VS, or make) after
> touching any CMake file. To give you an idea, we have about 1000 targets
> and that takes a good 2 min for CMake to rerun.
>
> Are there any plans to speed this up? Maybe parallelize it in some way or
> do a better job regenerating only what needs regenerating? Is there
> anything we can do on our side to reduce our regeneration times?
>
> For example, if using a VS generator, each directory in the source that
> has a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch
> one of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.
>
> Thanks for any help.
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake



-- 
J. Caleb Wherry
*Scientific Software Engineer*


http://www.calebwherry.com
+1 (615) 708-5651
calebwhe...@gmail.com
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

[CMake] CMake Project Generation Speedup

[Damian]

Hi all,

We are still in the process of switching our large Make-based build to
CMake. One of the issues we're running into is the time it takes to reparse
and regenerate the CMake project (whether ninja, VS, or make) after
touching any CMake file. To give you an idea, we have about 1000 targets
and that takes a good 2 min for CMake to rerun.

Are there any plans to speed this up? Maybe parallelize it in some way or
do a better job regenerating only what needs regenerating? Is there
anything we can do on our side to reduce our regeneration times?

For example, if using a VS generator, each directory in the source that has
a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch one
of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.

Thanks for any help.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake