Re: [cmake-developers] Need ideas/opinions on third party library management

[Florent Castelli]

Well, I tried upstreaming the new build scripts to some projects and it didn’t 
go well.
Some of the reasons I’ve heard of:
- Windows developpers don’t use CMake, they have project files on the 
repository.
  The CMake files for Windows will never be updated.
- I installed CMake 2.8.6 five years ago and I don’t want to update yet again!
  People relying on old versions is quite common and any attempt to raise the 
min
  version will be frowned upon (see the discussion in the LLVM mailing lists 
for example).
- We prefer to use autotools and don’t want to have to learn CMake.
  That’s fair. But also, no one likes to build an autotools backed project for 
Android or iOS.

Rewriting build scripts for all libraries isn’t trivial. But depending on the 
scale of your
organization, it might make sense to spend the time.
Google is doing something very similar by writing Blaze build scripts for the 
*world*.
They can leverage all their infrastructure for fast and correct builds then.

In a similar way here, we just have our compilation options set once in our 
toolchain
file and that’s it, it works correctly, without duplicating jobs to build the 
libraries.
Also, everything is built from the tip of the tree, which is a different 
software practice
than using set releases. It’s not for everyone though.

As for using a heterogenous build system, I’m really against it.
Even if they work for the platform you are trying to target, you may run into 
other issues.
Some build systems are really badly designed and will slowdown your iterations 
a lot.
Most people here will use Ninja and have a very fast or instant “no-op” build.
You may forget to keep options in sync. Or even might not be able to pass some 
options
to the other build system.

I’ve never heard of Spack before. It looks better than other solutions I’ve 
seen before.
But you still have to manage all the options from your build script and publish
the binaries somewhere. Then you need to teach your build scripts to get the 
right version.
I won’t trade my builds from source for a set of prebuilt binaries anytime soon 
I think :)


/Florent

> On 16 Aug 2016, at 14:52, Elizabeth A. Fischer 
>  wrote:
> 
> CMake builds for existing libraries are certainly an interesting and useful 
> thing, and deserve to be posted in a GitHub repo somewhere.  They should also 
> serve as the basis of a campaign to get the library authors to incorporate 
> the CMake build directly in their repos.
> 
> But any approach that requires every build to be ported to CMake will be 
> difficult and labor-prone to scale.  Writing a meta-build recipe is usually 
> much easier.
> 
> Spack handles the combinatorial dependencies you mention in a sophisticated, 
> graceful way that most meta-builders do not.  Its only problem is it does not 
> (yet) run on Windows.  There's no fundamental reason why not; we just need 
> someone to get involved and start trying it on Windows.
> 
> -- Elizabeth
> 
> 
> On Tue, Aug 16, 2016 at 6:52 AM, Florent Castelli  > wrote:
> At Spotify, we use CMake a lot for our large C++ library shared by all the 
> clients.
> After trying to build libraries for each platform and variant, we basically 
> gave up and we now
> use a super-build approach.
> 
> For example, Boost is used by 5 platforms: Windows, OSX, Linux, Android and 
> iOS.
> Each platform has a different CPU target (or many 32/64bit, x86/ARM).
> Each platform has many compilers.
> Some platforms have instrumentation options (Debug / Release, ASan, MSan…) 
> and really need
> to be compiled properly, otherwise you’ll end up with false positives.
> The matrix of builds is REALLY hard to track. Each time we update Boost, we 
> had to update
> a lot of things.
> I tried using ExternalProject and use b2 (build tool from Boost) to build it 
> and instead of having
> lots of build jobs with a mirror of the flags, you end up mirroring the flags 
> in your CMake files
> instead, which is still not good enough.
> 
> In the end, I looked at how Boost is actually built. And for most libraries, 
> it’s plain simple.
> A static library with a few files, some define, sometimes a platform specific 
> source file.
> What if instead of having an external build tool, I built it from CMake 
> instead?
> It would propagate all the build flags, target, instrumentation and compiler 
> information from the main
> build to it and just work.
> I tried it and it worked in no time! We replaced our Boost 1.59 binary 
> distribution with the source
> distribution and it’s much easier. When people build our library for a 
> different target, they don’t have
> to download new binaries, they just reuse the same sources.
> Later on, we found a bug in Boost 1.59 (fixed in later versions) and patched 
> it. We updated our source
> bundle and everything was smooth.
> Much later on, we wanted to use 1.61. We just updated the source bundle 
> again, the list of source
> files or compilation flags for the libra

Re: [cmake-developers] Need ideas/opinions on third party library management

[Florent Castelli]

We have many smaller libraries and a few big ones like Boost.
It certainly takes a bit of time to setup, but it will save you a lot of time 
later.
It will keep your build scripts clean too as the user won’t have to know about
its dependencies setup. Just use “target_link_libraries” on the modern target
that was created and it will work.

Most of the 3rd party dependencies don’t work that way and they're quite painful
to use correctly. As in, if you miss to add the “add_definitions” they told you 
to use,
it will build, but source files might be used with a different preprocessor 
define and
you end up with two different implementations of the same class in the same 
binary.
It will crash in very mysterious ways. 

The most important point is *correctness*. If you experiment with tooling like
the Clang sanitizers, you’ll know that you can’t mix libraries that haven’t been
instrumented with instrumented code.
Best case the libraries are self-contained and will work all of the time.
Worst case, the libraries are mostly self-contained and will *not* work 
sometimes!

As an example, the Boost Unit Test Framework usually worked fine for us.
But sometimes, we had ASan triggering an issue in the constructor of the string
object for the test name. Our binary was built with ASan and Boost without.
When we built it from source, all the false positives went away.

I know that not a lot of people use them or have heard of them, but they are
a super important tool that will find TONS of real issues in your code the first
time you use them and we cannot do without.
Some people might rely on Valgrind Memcheck instead, but hey, this is 2016
and we like our tests to run fast(er)!


/Florent

> On 16 Aug 2016, at 14:41, Benjamin Ballet  wrote:
> 
> Very interesting discussion, we have the same issues here.
> 
> Florent Castelli, how many third parties libraries do you use ? I think a 
> super build can be a very good solution but I'm wondering how much third 
> party code you have to build. Here we use OpenCV, with, boost, and poco, and 
> other things... So it may be too long.
> 
> I was personnaly considering having an hybrid solution : include small 
> libraries (like jsoncpp) and pre-build the other for each platforms.
> 
> 
> 2016-08-16 12:52 GMT+02:00 Florent Castelli  >:
> At Spotify, we use CMake a lot for our large C++ library shared by all the 
> clients.
> After trying to build libraries for each platform and variant, we basically 
> gave up and we now
> use a super-build approach.
> 
> For example, Boost is used by 5 platforms: Windows, OSX, Linux, Android and 
> iOS.
> Each platform has a different CPU target (or many 32/64bit, x86/ARM).
> Each platform has many compilers.
> Some platforms have instrumentation options (Debug / Release, ASan, MSan…) 
> and really need
> to be compiled properly, otherwise you’ll end up with false positives.
> The matrix of builds is REALLY hard to track. Each time we update Boost, we 
> had to update
> a lot of things.
> I tried using ExternalProject and use b2 (build tool from Boost) to build it 
> and instead of having
> lots of build jobs with a mirror of the flags, you end up mirroring the flags 
> in your CMake files
> instead, which is still not good enough.
> 
> In the end, I looked at how Boost is actually built. And for most libraries, 
> it’s plain simple.
> A static library with a few files, some define, sometimes a platform specific 
> source file.
> What if instead of having an external build tool, I built it from CMake 
> instead?
> It would propagate all the build flags, target, instrumentation and compiler 
> information from the main
> build to it and just work.
> I tried it and it worked in no time! We replaced our Boost 1.59 binary 
> distribution with the source
> distribution and it’s much easier. When people build our library for a 
> different target, they don’t have
> to download new binaries, they just reuse the same sources.
> Later on, we found a bug in Boost 1.59 (fixed in later versions) and patched 
> it. We updated our source
> bundle and everything was smooth.
> Much later on, we wanted to use 1.61. We just updated the source bundle 
> again, the list of source
> files or compilation flags for the libraries we use didn’t change. It was 
> again effortless.
> 
> Overall, building boost takes 10s on our developers’ machines. The sources 
> aren’t changed often,
> so the cost is pretty low. It needs attention when we upgrade it, but that’s 
> quite rare.
> 
> We try now to use the same approach for other libraries when we add them. 
> Some of them are
> already using CMake and it’s somewhat easier, but since most people still 
> target version 2.8 (or 2.6...),
> we find it better to rewrite the build scripts ourselves and use modern 
> features (as in, everything is
> a target that propagates requirements, we don’t propagate variables).
> It makes it also much easier to build a library for another platform that 
> wa

Re: [cmake-developers] Need ideas/opinions on third party library management

[Elizabeth A. Fischer]

CMake builds for existing libraries are certainly an interesting and useful
thing, and deserve to be posted in a GitHub repo somewhere.  They should
also serve as the basis of a campaign to get the library authors to
incorporate the CMake build directly in their repos.

But any approach that requires every build to be ported to CMake will be
difficult and labor-prone to scale.  Writing a meta-build recipe is usually
much easier.

Spack handles the combinatorial dependencies you mention in a
sophisticated, graceful way that most meta-builders do not.  Its only
problem is it does not (yet) run on Windows.  There's no fundamental reason
why not; we just need someone to get involved and start trying it on
Windows.

-- Elizabeth


On Tue, Aug 16, 2016 at 6:52 AM, Florent Castelli <
florent.caste...@gmail.com> wrote:

> At Spotify, we use CMake a lot for our large C++ library shared by all the
> clients.
> After trying to build libraries for each platform and variant, we
> basically gave up and we now
> use a super-build approach.
>
> For example, Boost is used by 5 platforms: Windows, OSX, Linux, Android
> and iOS.
> Each platform has a different CPU target (or many 32/64bit, x86/ARM).
> Each platform has many compilers.
> Some platforms have instrumentation options (Debug / Release, ASan, MSan…)
> and really need
> to be compiled properly, otherwise you’ll end up with false positives.
> The matrix of builds is REALLY hard to track. Each time we update Boost,
> we had to update
> a lot of things.
> I tried using ExternalProject and use b2 (build tool from Boost) to build
> it and instead of having
> lots of build jobs with a mirror of the flags, you end up mirroring the
> flags in your CMake files
> instead, which is still not good enough.
>
> In the end, I looked at how Boost is actually built. And for most
> libraries, it’s plain simple.
> A static library with a few files, some define, sometimes a platform
> specific source file.
> What if instead of having an external build tool, I built it from CMake
> instead?
> It would propagate all the build flags, target, instrumentation and
> compiler information from the main
> build to it and just work.
> I tried it and it worked in no time! We replaced our Boost 1.59 binary
> distribution with the source
> distribution and it’s much easier. When people build our library for a
> different target, they don’t have
> to download new binaries, they just reuse the same sources.
> Later on, we found a bug in Boost 1.59 (fixed in later versions) and
> patched it. We updated our source
> bundle and everything was smooth.
> Much later on, we wanted to use 1.61. We just updated the source bundle
> again, the list of source
> files or compilation flags for the libraries we use didn’t change. It was
> again effortless.
>
> Overall, building boost takes 10s on our developers’ machines. The sources
> aren’t changed often,
> so the cost is pretty low. It needs attention when we upgrade it, but
> that’s quite rare.
>
> We try now to use the same approach for other libraries when we add them.
> Some of them are
> already using CMake and it’s somewhat easier, but since most people still
> target version 2.8 (or 2.6...),
> we find it better to rewrite the build scripts ourselves and use modern
> features (as in, everything is
> a target that propagates requirements, we don’t propagate variables).
> It makes it also much easier to build a library for another platform that
> wasn’t targeted by the original
> project.
>
> If people are interested, I could share the CMakeLists.txt file we use for
> Boost. It doesn’t build all
> the libraries (some are hard like Context) and uses some internal macros,
> but it should be plain
> simple to tweak for your use.
>
> /Florent
>
> > On 12 Aug 2016, at 21:59, Robert Dailey 
> wrote:
> >
> > Hello,
> >
> > There is an internal C++ product at the company I work for which I
> > have written a series of CMake scripts for. This project actually has
> > dependencies on several open source libraries, such as boost,
> > freetype, openssl, etc.
> >
> > Right now what we do is build each of these third party libraries *by
> > hand*, once for every platform we support (Windows, Linux x86, Android
> > NDK). Then we stuff the includes (headers) and libraries
> > (static/shared) in a submodule and the primary code base's CMake
> > scripts pull them in as interface targets.
> >
> > This works well and is light-weight but is a pain when upgrading or
> > changing libraries. It's a pain because if I want to upgrade boost, I
> > have to build it up to 6 times (once for each platform and once for
> > each configuration).
> >
> > I've been thinking of a different approach for a while. I've done some
> > toying around with the "Super Build" concept, where I have a separate
> > CMake project that does nothing but use the ExternalProject module to
> > build libraries in real time along with our project. So the order of
> > operations would be as follows (for our automat

Re: [cmake-developers] Need ideas/opinions on third party library management

[Benjamin Ballet via cmake-developers]

Very interesting discussion, we have the same issues here.

Florent Castelli, how many third parties libraries do you use ? I think a
super build can be a very good solution but I'm wondering how much third
party code you have to build. Here we use OpenCV, with, boost, and poco,
and other things... So it may be too long.

I was personnaly considering having an hybrid solution : include small
libraries (like jsoncpp) and pre-build the other for each platforms.


2016-08-16 12:52 GMT+02:00 Florent Castelli :

> At Spotify, we use CMake a lot for our large C++ library shared by all the
> clients.
> After trying to build libraries for each platform and variant, we
> basically gave up and we now
> use a super-build approach.
>
> For example, Boost is used by 5 platforms: Windows, OSX, Linux, Android
> and iOS.
> Each platform has a different CPU target (or many 32/64bit, x86/ARM).
> Each platform has many compilers.
> Some platforms have instrumentation options (Debug / Release, ASan, MSan…)
> and really need
> to be compiled properly, otherwise you’ll end up with false positives.
> The matrix of builds is REALLY hard to track. Each time we update Boost,
> we had to update
> a lot of things.
> I tried using ExternalProject and use b2 (build tool from Boost) to build
> it and instead of having
> lots of build jobs with a mirror of the flags, you end up mirroring the
> flags in your CMake files
> instead, which is still not good enough.
>
> In the end, I looked at how Boost is actually built. And for most
> libraries, it’s plain simple.
> A static library with a few files, some define, sometimes a platform
> specific source file.
> What if instead of having an external build tool, I built it from CMake
> instead?
> It would propagate all the build flags, target, instrumentation and
> compiler information from the main
> build to it and just work.
> I tried it and it worked in no time! We replaced our Boost 1.59 binary
> distribution with the source
> distribution and it’s much easier. When people build our library for a
> different target, they don’t have
> to download new binaries, they just reuse the same sources.
> Later on, we found a bug in Boost 1.59 (fixed in later versions) and
> patched it. We updated our source
> bundle and everything was smooth.
> Much later on, we wanted to use 1.61. We just updated the source bundle
> again, the list of source
> files or compilation flags for the libraries we use didn’t change. It was
> again effortless.
>
> Overall, building boost takes 10s on our developers’ machines. The sources
> aren’t changed often,
> so the cost is pretty low. It needs attention when we upgrade it, but
> that’s quite rare.
>
> We try now to use the same approach for other libraries when we add them.
> Some of them are
> already using CMake and it’s somewhat easier, but since most people still
> target version 2.8 (or 2.6...),
> we find it better to rewrite the build scripts ourselves and use modern
> features (as in, everything is
> a target that propagates requirements, we don’t propagate variables).
> It makes it also much easier to build a library for another platform that
> wasn’t targeted by the original
> project.
>
> If people are interested, I could share the CMakeLists.txt file we use for
> Boost. It doesn’t build all
> the libraries (some are hard like Context) and uses some internal macros,
> but it should be plain
> simple to tweak for your use.
>
> /Florent
>
> > On 12 Aug 2016, at 21:59, Robert Dailey 
> wrote:
> >
> > Hello,
> >
> > There is an internal C++ product at the company I work for which I
> > have written a series of CMake scripts for. This project actually has
> > dependencies on several open source libraries, such as boost,
> > freetype, openssl, etc.
> >
> > Right now what we do is build each of these third party libraries *by
> > hand*, once for every platform we support (Windows, Linux x86, Android
> > NDK). Then we stuff the includes (headers) and libraries
> > (static/shared) in a submodule and the primary code base's CMake
> > scripts pull them in as interface targets.
> >
> > This works well and is light-weight but is a pain when upgrading or
> > changing libraries. It's a pain because if I want to upgrade boost, I
> > have to build it up to 6 times (once for each platform and once for
> > each configuration).
> >
> > I've been thinking of a different approach for a while. I've done some
> > toying around with the "Super Build" concept, where I have a separate
> > CMake project that does nothing but use the ExternalProject module to
> > build libraries in real time along with our project. So the order of
> > operations would be as follows (for our automated build server):
> >
> > 1. Clone our "Third Party" repository
> > 2. Use CMake to generate & build the "Super Build" project (this
> > builds boost, openssl, freetype, etc for the current platform).
> > 3. Clone the main code base's repository
> > 4. Use CMake to generate & build, using find_package() to ref

Re: [cmake-developers] Need ideas/opinions on third party library management

[Florent Castelli]

At Spotify, we use CMake a lot for our large C++ library shared by all the 
clients.
After trying to build libraries for each platform and variant, we basically 
gave up and we now
use a super-build approach.

For example, Boost is used by 5 platforms: Windows, OSX, Linux, Android and iOS.
Each platform has a different CPU target (or many 32/64bit, x86/ARM).
Each platform has many compilers.
Some platforms have instrumentation options (Debug / Release, ASan, MSan…) and 
really need
to be compiled properly, otherwise you’ll end up with false positives.
The matrix of builds is REALLY hard to track. Each time we update Boost, we had 
to update
a lot of things.
I tried using ExternalProject and use b2 (build tool from Boost) to build it 
and instead of having
lots of build jobs with a mirror of the flags, you end up mirroring the flags 
in your CMake files
instead, which is still not good enough.

In the end, I looked at how Boost is actually built. And for most libraries, 
it’s plain simple.
A static library with a few files, some define, sometimes a platform specific 
source file.
What if instead of having an external build tool, I built it from CMake instead?
It would propagate all the build flags, target, instrumentation and compiler 
information from the main
build to it and just work.
I tried it and it worked in no time! We replaced our Boost 1.59 binary 
distribution with the source
distribution and it’s much easier. When people build our library for a 
different target, they don’t have
to download new binaries, they just reuse the same sources.
Later on, we found a bug in Boost 1.59 (fixed in later versions) and patched 
it. We updated our source
bundle and everything was smooth.
Much later on, we wanted to use 1.61. We just updated the source bundle again, 
the list of source
files or compilation flags for the libraries we use didn’t change. It was again 
effortless.

Overall, building boost takes 10s on our developers’ machines. The sources 
aren’t changed often,
so the cost is pretty low. It needs attention when we upgrade it, but that’s 
quite rare.

We try now to use the same approach for other libraries when we add them. Some 
of them are
already using CMake and it’s somewhat easier, but since most people still 
target version 2.8 (or 2.6...),
we find it better to rewrite the build scripts ourselves and use modern 
features (as in, everything is
a target that propagates requirements, we don’t propagate variables).
It makes it also much easier to build a library for another platform that 
wasn’t targeted by the original
project.

If people are interested, I could share the CMakeLists.txt file we use for 
Boost. It doesn’t build all
the libraries (some are hard like Context) and uses some internal macros, but 
it should be plain
simple to tweak for your use.

/Florent

> On 12 Aug 2016, at 21:59, Robert Dailey  wrote:
> 
> Hello,
> 
> There is an internal C++ product at the company I work for which I
> have written a series of CMake scripts for. This project actually has
> dependencies on several open source libraries, such as boost,
> freetype, openssl, etc.
> 
> Right now what we do is build each of these third party libraries *by
> hand*, once for every platform we support (Windows, Linux x86, Android
> NDK). Then we stuff the includes (headers) and libraries
> (static/shared) in a submodule and the primary code base's CMake
> scripts pull them in as interface targets.
> 
> This works well and is light-weight but is a pain when upgrading or
> changing libraries. It's a pain because if I want to upgrade boost, I
> have to build it up to 6 times (once for each platform and once for
> each configuration).
> 
> I've been thinking of a different approach for a while. I've done some
> toying around with the "Super Build" concept, where I have a separate
> CMake project that does nothing but use the ExternalProject module to
> build libraries in real time along with our project. So the order of
> operations would be as follows (for our automated build server):
> 
> 1. Clone our "Third Party" repository
> 2. Use CMake to generate & build the "Super Build" project (this
> builds boost, openssl, freetype, etc for the current platform).
> 3. Clone the main code base's repository
> 4. Use CMake to generate & build, using find_package() to refer to
> interface targets exported by those third party libraries built in
> step 2
> 
> Obviously this will make builds take significantly longer, because
> we're constantly rebuilding the same third party libraries over and
> over again. However, it virtually eliminates the maintenance burden
> for third party libraries because they are built inherently with
> everything else.
> 
> Note that I can't refer to pre-built libraries in our build
> environments because we need very specific control over the versions
> of our libraries as well as the toolchains that were used to build
> them. Also we may specifically build our libraries a certain way (such
> as boost). For t

[cmake-developers] Need ideas/opinions on third party library management

[Robert Dailey]

Hello,

There is an internal C++ product at the company I work for which I
have written a series of CMake scripts for. This project actually has
dependencies on several open source libraries, such as boost,
freetype, openssl, etc.

Right now what we do is build each of these third party libraries *by
hand*, once for every platform we support (Windows, Linux x86, Android
NDK). Then we stuff the includes (headers) and libraries
(static/shared) in a submodule and the primary code base's CMake
scripts pull them in as interface targets.

This works well and is light-weight but is a pain when upgrading or
changing libraries. It's a pain because if I want to upgrade boost, I
have to build it up to 6 times (once for each platform and once for
each configuration).

I've been thinking of a different approach for a while. I've done some
toying around with the "Super Build" concept, where I have a separate
CMake project that does nothing but use the ExternalProject module to
build libraries in real time along with our project. So the order of
operations would be as follows (for our automated build server):

1. Clone our "Third Party" repository
2. Use CMake to generate & build the "Super Build" project (this
builds boost, openssl, freetype, etc for the current platform).
3. Clone the main code base's repository
4. Use CMake to generate & build, using find_package() to refer to
interface targets exported by those third party libraries built in
step 2

Obviously this will make builds take significantly longer, because
we're constantly rebuilding the same third party libraries over and
over again. However, it virtually eliminates the maintenance burden
for third party libraries because they are built inherently with
everything else.

Note that I can't refer to pre-built libraries in our build
environments because we need very specific control over the versions
of our libraries as well as the toolchains that were used to build
them. Also we may specifically build our libraries a certain way (such
as boost). For this reason we do not rely on our external environment
or external package managers to fulfill third party dependencies, like
most open source projects do on Linux for example.

Does this "Super Build" approach sound like a better idea? What other
options are available? The downside with the "Super Build" solution is
that it will become very difficult to make the transition between
building third party and building our code base seamless. I can't do
both in the same generate step because find_package() can't be called
until the libraries are built & installed.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers