> > Overall, the changes are extremely small. We (apparently) did a pretty > good job of making the CMake configuration scripts modular and > consumable by arbitrary projects
ah, the warm, fuzzy feeling that one experiences when all the hard work of abstracting stuff pays off :-) well done, Alex! A quick question for you (as you recall, I have great stakes in - but minimal understanding of - cmake) - which targets should be expected to work for Cmake? I had to add a few (minor) fixes to make CLion fully understand Mesos (it still has a few "bogus" errors, but it's by and large, greatly usable - and beats Eclipse CDT any day). Also, please let me know whether there's anything you'd like me to try out and report back. Thanks! -- *Marco Massenzio* http://codetrips.com On Fri, Feb 26, 2016 at 11:44 AM, Alex Clemmer <clemmer.alexan...@gmail.com> wrote: > Ah, I see now that I coudl have done better by splitting the patch > into two patches -- one where we move everything, and one where we > change the cmake files. > > THe important parts of the patch are: > > * Adding the contents of `3rdparty/libprocess/3rdparty/CMakeLists.txt` > -> `3rdparty/CMakeLists.txt`; makes since, because we're merging the > two 3rdparty folders, so we only need one CMakeLists.txt. We just dump > the contents unchanged into the other one. > * `3rdparty/libprocess/cmake/Process3rdpartyConfigure.cmake` changing some > paths > * `3rdparty/libprocess/CMakeLists.txt` moving a call to `include`. > > And that's about it. Everything else is just moving. > > On Fri, Feb 26, 2016 at 11:36 AM, Alex Clemmer > <clemmer.alexan...@gmail.com> wrote: > > Folks: > > > > Took about 30 minutes to put together a prototype of the great > > `3rdparty` flattening. Please see the (extremely preliminary) > > review[1] or my working branch[2] for a really close approximation of > > the number of changes to the CMake build system we'd need to support > > this flattening. > > > > Overall, the changes are extremely small. We (apparently) did a pretty > > good job of making the CMake configuration scripts modular and > > consumable by arbitrary projects, so the changes are mostly things > > like "move the config `include` call over there instead of being over > > there" and "change a handful of path variables to reflect the new dir > > structure". (I hope, btw, that it doesn't seem arrogant to say this, > > but I think it is justified given the relatively minor changes in the > > actual CMake code.) > > > > Feel free to remix/take/throw away any subset of this. I don't need > > any credit for it in the final patch if someone marches off in that > > direction. :) > > > > [1] https://reviews.apache.org/r/44099/ > > [2] https://github.com/hausdorff/mesos/commits/prototype_flattened_3rd > > > > On Thu, Feb 18, 2016 at 12:22 PM, Kevin Klues <klue...@gmail.com> wrote: > >> I am also a fan of git submodules in the long term, but avoiding them > >> in the short term. We should get things organized as we want them > >> first, and then start thinking about pulling libprocess/stout out into > >> submodules later (while also preserving their history!) > >> > >> I disagree with moving libprocess and stout up to the same level as > >> src/. If we want to make sure they don't bleed into Mesos proper, we > >> really should treat them the same as any other 3rdparty code that we > >> depend on. This will become more relevant when/if we move them into > >> submodules. > >> > >> Given all that, the only real challenge with flattening our 3rdparty > >> dependencies into a single folder should be the changes we have to > >> make to our configure.ac and Makefile.am scripts to know where to look > >> for their dependencies now. In the end these should be URLs to > >> versioned tarballs that we host somewhere (or git repos that we can > >> have forked and tagged with specific versions). In the short term > >> these can just be relative paths in the mesos tree though. > >> > >> On Tue, Feb 16, 2016 at 1:26 PM, Kapil Arya <ka...@mesosphere.io> > wrote: > >>> Thanks for bringing it up Alexander! > >>> > >>> I don't have a strong opinion wrt git submodules since I don't have > >>> much experience with them personally. Having said that, I would like > >>> to go conservative on this one (baby steps :-) ). > >>> > >>> Further, I do understand that moving libprocess and stout directories > >>> will be painful for people who already have several branches and will > >>> have conflicts. But I do think, there are some interim solutions as > >>> well (for example, move libprocess/stout to wherever we want, but keep > >>> a symlink from 3rdparty/libprocess, etc, to those new locations for > >>> some time). I am sure there are better solutions out there, but this > >>> should work too. > >>> > >>> Best, > >>> Kapil > >>> > >>> On Tue, Feb 16, 2016 at 12:38 PM, Erik Weathers > >>> <eweath...@groupon.com.invalid> wrote: > >>>> If we go to git submodules, please ensure there are good docs around > how to > >>>> update cloned repos. > >>>> > >>>> e.g., From ansible: > https://docs.ansible.com/ansible/intro_installation.html > >>>> > >>>> Note when updating ansible, be sure to not only update the source > tree, but > >>>> also the “submodules” in git which point at Ansible’s own modules > (not the > >>>> same kind of modules, alas). > >>>> > >>>> $ git pull --rebase > >>>> $ git submodule update --init --recursive > >>>> > >>>> Thanks, > >>>> > >>>> - Erik > >>>> > >>>> On Tue, Feb 16, 2016 at 8:54 AM, Alexander Rojas < > alexan...@mesosphere.io> > >>>> wrote: > >>>> > >>>>> +1 > >>>>> I am one who is totally in for that change. It is not only the > directories > >>>>> problem, but the structure which has led that the stout tests (which > do > >>>>> need to be compiled) are actually managed in the libprocess > Makefile, on > >>>>> top of all the things you have already mentioned. > >>>>> > >>>>> > >>>>> > On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io> wrote: > >>>>> > > >>>>> > On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yujie....@gmail.com> > wrote: > >>>>> >> Kapil, > >>>>> >> > >>>>> >> I guess what I want to understand is why the existing structure > makes it > >>>>> >> hard for you to do the things that you want to do (installing > >>>>> >> module-specific 3rdparty dependencies into > "${pkglibdir}/3rdparty" as > >>>>> part > >>>>> >> of "make install"). > >>>>> > > >>>>> > Let me see if I can answer that :-). > >>>>> > > >>>>> > This is somewhat related. For example, if we want to install > protobuf > >>>>> > in 3rdparty/{include,lib} (for module developers to use them > without > >>>>> > doing a proper mesos installation), you need to provide the correct > >>>>> > "--prefix" flag that points to 3rdparty/. However, due to multiple > >>>>> > levels of configure.ac, the "--prefix" can at best be generated by > >>>>> > prepending "../../../" to get to the great-grandparent directory. > This > >>>>> > is because we have a separate configure.ac which manages > >>>>> > 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it, > >>>>> > but they are not clean. > >>>>> > > >>>>> > Similar thing holds for system-wide installation of these 3rdparty > >>>>> > packages. For example, ideally, we would want to use > >>>>> > "${pkglibdir}/3rdparty" as a prefix for those packages. However, > since > >>>>> > they are part of libprocess package, we don't get the correct > >>>>> > directory and have to use either hardwired $pkglibdir, or somehow > pass > >>>>> > it from the top-level configure all the way down to > >>>>> > 3rdparty/libprocess/3rdparty/Makefile.am :-(. > >>>>> > > >>>>> > > >>>>> >> The only reason you mentioned in the original email is that "in > the > >>>>> current > >>>>> >> code base, we don't strictly follow the 3rdparty structure", > which IMO > >>>>> is > >>>>> >> not a very convincing reason for such a big change. > >>>>> > > >>>>> > How about a not so big change? :-). What if we just move > >>>>> > 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving > >>>>> > stout as is? That is not a big change since we are not touching > >>>>> > libprocess/stout. Just adjusting Makefiles and I am pretty sure it > >>>>> > will be cleaner and simpler than what we have right now. > >>>>> > > >>>>> > As a later time, we can then consider moving stout out to 3rdparty/ > >>>>> > while leaving libprocess as is. But that's something we can decide > >>>>> > later and leave stout as an exception for now. > >>>>> > > >>>>> > BTW, if we were to install all the 3rdparty packages in 3rdparty/, > >>>>> > that would also cut down a lot on the compiler flags (i.e., fewer > "-I" > >>>>> > and "-L" flags) :-). > >>>>> > > >>>>> > Kapil > >>>>> > > >>>>> >> > >>>>> >> - Jie > >>>>> >> > >>>>> >> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> > wrote: > >>>>> >> > >>>>> >>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yujie....@gmail.com> > wrote: > >>>>> >>>> > >>>>> >>>>> > >>>>> >>>>> However, in the current code base, we don't strictly follow the > >>>>> >>> 3rdparty > >>>>> >>>>> structure. For example, stout has a dependency on picojson and > >>>>> >>>>> google-protobuf, but we don't put these two packages inside > >>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/. > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> My understanding is that stout is header only. So it does not > have to > >>>>> >>>> bundle 3rdparty libraries. The user of stout is responsible for > >>>>> bundling > >>>>> >>>> them if they are used. > >>>>> >>> > >>>>> >>> > >>>>> >>> I don't think being header-only is an excuse to have a broken > >>>>> >>> installation :-). Further, we don't make it easier for the user > to get > >>>>> >>> the 3rdparty binaries either. For example, if the user has a > different > >>>>> >>> version of protobuf installed on the system, the compilation of > any > >>>>> >>> program that uses stout will fail spectacularly! > >>>>> >>> > >>>>> >>> Having said that, the gist here is that we have somewhat > deviated from > >>>>> >>> original motivation behind the 3rdparty directory and it would > be nice > >>>>> >>> if we can have a flatter structure. > >>>>> >>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> - Jie > >>>>> >>>> > >>>>> >>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io > > > >>>>> wrote: > >>>>> >>>> > >>>>> >>>>> Hi All, > >>>>> >>>>> > >>>>> >>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into > >>>>> >>> 3rdparty/. > >>>>> >>>>> (Optionally) Move libprocess/stout to the top-level directory. > >>>>> >>>>> > >>>>> >>>>> I wanted to start some discussion around reorganizing stuff > inside > >>>>> >>>>> "3rdparty". I apologize for the length of the email, please > bear with > >>>>> >>> me. > >>>>> >>>>> > >>>>> >>>>> Traditionally, 3rdparty has been used to hold all Mesos > dependencies > >>>>> >>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further, > >>>>> >>>>> libprocess/3rdparty was to hold all libprocess dependencies > (which > >>>>> may > >>>>> >>> in > >>>>> >>>>> turn be Mesos dependencies as well). > >>>>> >>>>> > >>>>> >>>>> As I understand, the original motivation was to emphasize that > >>>>> >>> libprocess > >>>>> >>>>> is an independent project which depends on "stout", which in > turn is > >>>>> >>> also > >>>>> >>>>> an independent project. > >>>>> >>>>> > >>>>> >>>>> However, in the current code base, we don't strictly follow the > >>>>> >>> 3rdparty > >>>>> >>>>> structure. For example, stout has a dependency on picojson and > >>>>> >>>>> google-protobuf, but we don't put these two packages inside > >>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/. > >>>>> >>>>> > >>>>> >>>>> In light of these anomalies, I want to propose that we flatten > out > >>>>> the > >>>>> >>>>> 3rdparty directory and put all packages (libprocess, stout, > protobuf, > >>>>> >>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We > can > >>>>> still > >>>>> >>> use > >>>>> >>>>> "--with-XYZ=..." to the full extent as needed. > >>>>> >>>>> > >>>>> >>>>> To take it a step further, I want to propose that we bring > libprocess > >>>>> >>> and > >>>>> >>>>> stout out of 3rdparty/ and move them at the top level (i.e., > make > >>>>> them > >>>>> >>>>> peers of src/). That way, all code in 3rdparty/ is stuff from > "third" > >>>>> >>>>> parties and is used only when "--with-bundled" is defined (by > >>>>> default). > >>>>> >>>>> This hierarchy will still allow us to keep libprocess and > stout as > >>>>> >>> separate > >>>>> >>>>> independent projects. > >>>>> >>>>> > >>>>> >>>>> The motivation for this proposal came when dealing with > 3rdparty > >>>>> >>>>> dependencies for module development. A module developer needs > access > >>>>> to > >>>>> >>>>> protobuf, picojson, glog, etc., and for that matter, the exact > >>>>> >>> versions of > >>>>> >>>>> these packages that Mesos was compiled with. > >>>>> >>>>> > >>>>> >>>>> We want to solve this problem by installing module-specific > 3rdparty > >>>>> >>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make > install" > >>>>> (if > >>>>> >>>>> configured with something like > >>>>> "--enable-install-module-dependencies"). > >>>>> >>>>> (There is a discussion going on in a separate thread). > >>>>> >>>>> > >>>>> >>>>> Further, as of today, when we install Mesos using "make > install", we > >>>>> >>>>> install stout headers in "${prefix}/include/stout". However, > stout > >>>>> has > >>>>> >>>>> dependencies on picojson[1] and google-protobuf headers which > may not > >>>>> >>> be > >>>>> >>>>> present on the machine. This leaves stout, and in turn > libprocess and > >>>>> >>> Mesos > >>>>> >>>>> headers, fairly broken. I understand that this issue is > somewhat > >>>>> >>> orthogonal > >>>>> >>>>> to the main issue being discussed in this mail, but I wanted > to put > >>>>> it > >>>>> >>> out > >>>>> >>>>> since it's related. > >>>>> >>>>> > >>>>> >>>>> Any thoughts, comments, concerns are most welcome! > >>>>> >>>>> > >>>>> >>>>> Best, > >>>>> >>>>> Kapil > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> [1]: Picojson issue was resolved as part of > >>>>> >>>>> https://reviews.apache.org/r/41424/ which installs picojson.h > into > >>>>> the > >>>>> >>>>> include-dir. > >>>>> >>>>> > >>>>> >>> > >>>>> > >>>>> > >> > >> > >> > >> -- > >> ~Kevin > > > > > > > > -- > > Alex > > > > Theory is the first term in the Taylor series of practice. -- Thomas M > > Cover (1992) > > > > -- > Alex > > Theory is the first term in the Taylor series of practice. -- Thomas M > Cover (1992) >
