Re: [cmake-developers] cmake install command expected performance?

[Robert Goulet]

Great, let us know if the test is clean! Cheers!

-Original Message-
From: Brad King [mailto:brad.k...@kitware.com] 
Sent: Thursday, February 26, 2015 3:13 PM
To: Robert Goulet; Joshua Clayton
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] cmake install command expected performance?

On 02/26/2015 02:18 PM, Robert Goulet wrote:
> turns out to be 5000+ file open/close

Good catch!  It looks like it's been that way since manifest generation was 
first added here:

 ENH: When installing project, write manifest
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=99feab35

I've applied your patch with a few tweaks to avoid a leading newline and 
improve code formatting:

 install: Write the entire installation manifest at once
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=c4814174

> Please consider it for CMake 3.2!

Normally we don't take non-regression, non-new-feature-bugfix changes during a 
release candidate cycle.  Given your antivirus config one could consider this 
an environmental regression.  I'll consider it if it tests cleanly.

Thanks,
-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake install command expected performance?

[Brad King]

On 02/26/2015 02:18 PM, Robert Goulet wrote:
> turns out to be 5000+ file open/close

Good catch!  It looks like it's been that way since manifest generation
was first added here:

 ENH: When installing project, write manifest
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=99feab35

I've applied your patch with a few tweaks to avoid a leading
newline and improve code formatting:

 install: Write the entire installation manifest at once
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=c4814174

> Please consider it for CMake 3.2!

Normally we don't take non-regression, non-new-feature-bugfix changes
during a release candidate cycle.  Given your antivirus config one
could consider this an environmental regression.  I'll consider it
if it tests cleanly.

Thanks,
-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake install command expected performance?

[Robert Goulet]

Here is what we discovered. It looks like that when cmake write the install 
manifest file, it does it in a for_loop for each file, which in the end turns 
out to be 5000+ file open/close, and then our administrator controlled 
anti-virus checks the file each time, considerably slowing everything down.

We just changed the logic to write the install manifest file in a single file 
open/close rather than for each file, and it completely fixes our issue; it's 
now back to normal install times, now takes less than a second compared to 50+ 
seconds previously.

Patch file attached.
Please consider it for CMake 3.2!
Thanks!


-Original Message-
From: Brad King [mailto:brad.k...@kitware.com] 
Sent: Tuesday, February 24, 2015 10:07 AM
To: Robert Goulet; Joshua Clayton
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] cmake install command expected performance?

On 02/24/2015 10:03 AM, Robert Goulet wrote:
> Yes comparing timestamp is the way to go, but why is it so slow in cmake?

I've never observed that being slow in practice.

You'll have to profile CMake while running in this case, or add some print 
statements with high-precision timestamps to see where the time is going.

-Brad



patch_install_manifest_optimize.patch
Description: patch_install_manifest_optimize.patch
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake install command expected performance?

[Robert Goulet]

Ok I guess we have no other choice than go in and profile it then. Thanks for 
the input.

-Original Message-
From: Brad King [mailto:brad.k...@kitware.com] 
Sent: Tuesday, February 24, 2015 10:07 AM
To: Robert Goulet; Joshua Clayton
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] cmake install command expected performance?

On 02/24/2015 10:03 AM, Robert Goulet wrote:
> Yes comparing timestamp is the way to go, but why is it so slow in cmake?

I've never observed that being slow in practice.

You'll have to profile CMake while running in this case, or add some print 
statements with high-precision timestamps to see where the time is going.

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake install command expected performance?

[Brad King]

On 02/24/2015 10:03 AM, Robert Goulet wrote:
> Yes comparing timestamp is the way to go, but why is it so slow in cmake?

I've never observed that being slow in practice.

You'll have to profile CMake while running in this case, or add
some print statements with high-precision timestamps to see
where the time is going.

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake install command expected performance?

[Robert Goulet]

Yes comparing timestamp is the way to go, but why is it so slow in cmake? I 
added command to set it to lazy report and it saved 1 second. So it's something 
else... I forgot to mention that there's a weird pause at the end after 
installing all files... not sure what it's doing. It looks like this part is 
taking the most amount of time.

-Original Message-
From: Brad King [mailto:brad.k...@kitware.com] 
Sent: Tuesday, February 24, 2015 9:57 AM
To: Robert Goulet; Joshua Clayton
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] cmake install command expected performance?

On 02/24/2015 09:51 AM, Robert Goulet wrote:
> takes 50+ seconds for cmake to realize that all files don't need to be 
> copied

It's comparing the time stamps of every corresponding file.
On many filesystems that is faster than the actual copy.

> Perhaps the log spam for every file in the console is part of the problem?
> Is there a way to turn it off?

http://www.cmake.org/cmake/help/v3.1/variable/CMAKE_INSTALL_MESSAGE.html

 set(CMAKE_INSTALL_MESSAGE LAZY)

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake install command expected performance?

[Brad King]

On 02/24/2015 09:51 AM, Robert Goulet wrote:
> takes 50+ seconds for cmake to realize that all files don't need
> to be copied

It's comparing the time stamps of every corresponding file.
On many filesystems that is faster than the actual copy.

> Perhaps the log spam for every file in the console is part of the problem?
> Is there a way to turn it off?

http://www.cmake.org/cmake/help/v3.1/variable/CMAKE_INSTALL_MESSAGE.html

 set(CMAKE_INSTALL_MESSAGE LAZY)

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake install command expected performance?

[Robert Goulet]

Thanks everyone for all the hints! After a closer look, cmake actual copying of 
files is indeed just as fast as normal OS copy, tested after reboots everytime. 
However, when the files are already present (all target up-to-date), it still 
takes 50+ seconds for cmake to realize that all files don't need to be copied, 
compared to an OS copy or a robocopy in ruby that takes less than a second. 
Perhaps the log spam for every file in the console is part of the problem? Is 
there a way to turn it off?

-Original Message-
From: Joshua Clayton [mailto:stillcompil...@gmail.com] 
Sent: Monday, February 23, 2015 9:28 PM
To: Robert Goulet
Cc: Nils Gladitz; cmake-developers@cmake.org
Subject: Re: [cmake-developers] cmake install command expected performance?

On Linux reading the input file would still be cached on subsequent runs even  
if you remove the destination folder, and writing will always cache unless you 
have a filesystem mounted with -o sync (that supports sync) Without special 
effort, only the first time you run will it be reading from actual files on 
disk.
running the sync command immediately afterward can give you an idea how long it 
took to actually write out the files to media.

Not sure how this works on other OS's, but expect it is similar.

On Mon, Feb 23, 2015 at 8:10 AM, Robert Goulet  
wrote:
> We deleted the destination folder every time we did a test. And it's an SSD 
> drive, I don't think having files in cache would be the issue. When files are 
> already there it still takes 30+ seconds for cmake to "not" copy the files.
>
> -Original Message-
> From: Nils Gladitz [mailto:nilsglad...@gmail.com]
> Sent: Monday, February 23, 2015 10:37 AM
> To: Robert Goulet; cmake-developers@cmake.org
> Subject: Re: [cmake-developers] cmake install command expected performance?
>
> On 02/23/2015 04:13 PM, Robert Goulet wrote:
>> I'm running into an issue were using the CMake install command to 
>> copy a directory with over 5000+ files (slightly over 1GB of data) 
>> takes a lot more time than doing a simple copy. On an SSD drive it 
>> takes 0.2 seconds to copy, while it takes about 56+ seconds for CMake 
>> to do the same thing using the install command. Why is there so much 
>> a big difference? Is this intended for some reasons or could there be a bug? 
>> Thanks!
>
> Did you run the manual copy right after the cmake install? (e.g. could files 
> still have been in the system's disk cache?).
>
> Both a manual copy and a cmake install might omit copying if the source isn't 
> newer than the destination file; did you perform both with the destination 
> files missing?
>
> Nils
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more 
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake-developers
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake install command expected performance?

[Joshua Clayton]

On Linux reading the input file would still be cached on subsequent
runs even  if you remove the destination folder, and writing will
always cache unless you have a filesystem mounted with -o sync (that
supports sync)
Without special effort, only the first time you run will it be reading
from actual files on disk.
running the sync command immediately afterward can give you an idea
how long it took to actually write out the files to media.

Not sure how this works on other OS's, but expect it is similar.

On Mon, Feb 23, 2015 at 8:10 AM, Robert Goulet
 wrote:
> We deleted the destination folder every time we did a test. And it's an SSD 
> drive, I don't think having files in cache would be the issue. When files are 
> already there it still takes 30+ seconds for cmake to "not" copy the files.
>
> -Original Message-
> From: Nils Gladitz [mailto:nilsglad...@gmail.com]
> Sent: Monday, February 23, 2015 10:37 AM
> To: Robert Goulet; cmake-developers@cmake.org
> Subject: Re: [cmake-developers] cmake install command expected performance?
>
> On 02/23/2015 04:13 PM, Robert Goulet wrote:
>> I'm running into an issue were using the CMake install command to copy
>> a directory with over 5000+ files (slightly over 1GB of data) takes a
>> lot more time than doing a simple copy. On an SSD drive it takes 0.2
>> seconds to copy, while it takes about 56+ seconds for CMake to do the
>> same thing using the install command. Why is there so much a big
>> difference? Is this intended for some reasons or could there be a bug? 
>> Thanks!
>
> Did you run the manual copy right after the cmake install? (e.g. could files 
> still have been in the system's disk cache?).
>
> Both a manual copy and a cmake install might omit copying if the source isn't 
> newer than the destination file; did you perform both with the destination 
> files missing?
>
> Nils
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more 
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake-developers
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake install command expected performance?

[Robert Goulet]

We deleted the destination folder every time we did a test. And it's an SSD 
drive, I don't think having files in cache would be the issue. When files are 
already there it still takes 30+ seconds for cmake to "not" copy the files.

-Original Message-
From: Nils Gladitz [mailto:nilsglad...@gmail.com] 
Sent: Monday, February 23, 2015 10:37 AM
To: Robert Goulet; cmake-developers@cmake.org
Subject: Re: [cmake-developers] cmake install command expected performance?

On 02/23/2015 04:13 PM, Robert Goulet wrote:
> I'm running into an issue were using the CMake install command to copy 
> a directory with over 5000+ files (slightly over 1GB of data) takes a 
> lot more time than doing a simple copy. On an SSD drive it takes 0.2 
> seconds to copy, while it takes about 56+ seconds for CMake to do the 
> same thing using the install command. Why is there so much a big 
> difference? Is this intended for some reasons or could there be a bug? Thanks!

Did you run the manual copy right after the cmake install? (e.g. could files 
still have been in the system's disk cache?).

Both a manual copy and a cmake install might omit copying if the source isn't 
newer than the destination file; did you perform both with the destination 
files missing?

Nils
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake install command expected performance?

[Nils Gladitz]

On 02/23/2015 04:13 PM, Robert Goulet wrote:

I’m running into an issue were using the CMake install command to copy a
directory with over 5000+ files (slightly over 1GB of data) takes a lot
more time than doing a simple copy. On an SSD drive it takes 0.2 seconds
to copy, while it takes about 56+ seconds for CMake to do the same thing
using the install command. Why is there so much a big difference? Is
this intended for some reasons or could there be a bug? Thanks!


Did you run the manual copy right after the cmake install? (e.g. could 
files still have been in the system's disk cache?).


Both a manual copy and a cmake install might omit copying if the source 
isn't newer than the destination file; did you perform both with the 
destination files missing?


Nils
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] cmake install command expected performance?

[Robert Goulet]

Hi all,

I'm running into an issue were using the CMake install command to copy a 
directory with over 5000+ files (slightly over 1GB of data) takes a lot more 
time than doing a simple copy. On an SSD drive it takes 0.2 seconds to copy, 
while it takes about 56+ seconds for CMake to do the same thing using the 
install command. Why is there so much a big difference? Is this intended for 
some reasons or could there be a bug? Thanks!



Robert Goulet
Senior Software Developer
Games Solutions Group
Autodesk Media & Entertainment

MAIN +1 514 393 1616
DIRECT +1 514 954 3911
MOBILE +1 438 397 4075

Autodesk, Inc.
10 Rue Duke
Montreal, QC, H3C 2L7
www.autodesk.com

[Description: Email_Signature_Logobar]

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers