Ah, I see now that I coudl have done better by splitting the patch into two patches -- one where we move everything, and one where we change the cmake files.
THe important parts of the patch are: * Adding the contents of `3rdparty/libprocess/3rdparty/CMakeLists.txt` -> `3rdparty/CMakeLists.txt`; makes since, because we're merging the two 3rdparty folders, so we only need one CMakeLists.txt. We just dump the contents unchanged into the other one. * `3rdparty/libprocess/cmake/Process3rdpartyConfigure.cmake` changing some paths * `3rdparty/libprocess/CMakeLists.txt` moving a call to `include`. And that's about it. Everything else is just moving. On Fri, Feb 26, 2016 at 11:36 AM, Alex Clemmer <clemmer.alexan...@gmail.com> wrote: > Folks: > > Took about 30 minutes to put together a prototype of the great > `3rdparty` flattening. Please see the (extremely preliminary) > review[1] or my working branch[2] for a really close approximation of > the number of changes to the CMake build system we'd need to support > this flattening. > > Overall, the changes are extremely small. We (apparently) did a pretty > good job of making the CMake configuration scripts modular and > consumable by arbitrary projects, so the changes are mostly things > like "move the config `include` call over there instead of being over > there" and "change a handful of path variables to reflect the new dir > structure". (I hope, btw, that it doesn't seem arrogant to say this, > but I think it is justified given the relatively minor changes in the > actual CMake code.) > > Feel free to remix/take/throw away any subset of this. I don't need > any credit for it in the final patch if someone marches off in that > direction. :) > > [1] https://reviews.apache.org/r/44099/ > [2] https://github.com/hausdorff/mesos/commits/prototype_flattened_3rd > > On Thu, Feb 18, 2016 at 12:22 PM, Kevin Klues <klue...@gmail.com> wrote: >> I am also a fan of git submodules in the long term, but avoiding them >> in the short term. We should get things organized as we want them >> first, and then start thinking about pulling libprocess/stout out into >> submodules later (while also preserving their history!) >> >> I disagree with moving libprocess and stout up to the same level as >> src/. If we want to make sure they don't bleed into Mesos proper, we >> really should treat them the same as any other 3rdparty code that we >> depend on. This will become more relevant when/if we move them into >> submodules. >> >> Given all that, the only real challenge with flattening our 3rdparty >> dependencies into a single folder should be the changes we have to >> make to our configure.ac and Makefile.am scripts to know where to look >> for their dependencies now. In the end these should be URLs to >> versioned tarballs that we host somewhere (or git repos that we can >> have forked and tagged with specific versions). In the short term >> these can just be relative paths in the mesos tree though. >> >> On Tue, Feb 16, 2016 at 1:26 PM, Kapil Arya <ka...@mesosphere.io> wrote: >>> Thanks for bringing it up Alexander! >>> >>> I don't have a strong opinion wrt git submodules since I don't have >>> much experience with them personally. Having said that, I would like >>> to go conservative on this one (baby steps :-) ). >>> >>> Further, I do understand that moving libprocess and stout directories >>> will be painful for people who already have several branches and will >>> have conflicts. But I do think, there are some interim solutions as >>> well (for example, move libprocess/stout to wherever we want, but keep >>> a symlink from 3rdparty/libprocess, etc, to those new locations for >>> some time). I am sure there are better solutions out there, but this >>> should work too. >>> >>> Best, >>> Kapil >>> >>> On Tue, Feb 16, 2016 at 12:38 PM, Erik Weathers >>> <eweath...@groupon.com.invalid> wrote: >>>> If we go to git submodules, please ensure there are good docs around how to >>>> update cloned repos. >>>> >>>> e.g., From ansible: >>>> https://docs.ansible.com/ansible/intro_installation.html >>>> >>>> Note when updating ansible, be sure to not only update the source tree, but >>>> also the “submodules” in git which point at Ansible’s own modules (not the >>>> same kind of modules, alas). >>>> >>>> $ git pull --rebase >>>> $ git submodule update --init --recursive >>>> >>>> Thanks, >>>> >>>> - Erik >>>> >>>> On Tue, Feb 16, 2016 at 8:54 AM, Alexander Rojas <alexan...@mesosphere.io> >>>> wrote: >>>> >>>>> +1 >>>>> I am one who is totally in for that change. It is not only the directories >>>>> problem, but the structure which has led that the stout tests (which do >>>>> need to be compiled) are actually managed in the libprocess Makefile, on >>>>> top of all the things you have already mentioned. >>>>> >>>>> >>>>> > On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io> wrote: >>>>> > >>>>> > On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yujie....@gmail.com> wrote: >>>>> >> Kapil, >>>>> >> >>>>> >> I guess what I want to understand is why the existing structure makes >>>>> >> it >>>>> >> hard for you to do the things that you want to do (installing >>>>> >> module-specific 3rdparty dependencies into "${pkglibdir}/3rdparty" as >>>>> part >>>>> >> of "make install"). >>>>> > >>>>> > Let me see if I can answer that :-). >>>>> > >>>>> > This is somewhat related. For example, if we want to install protobuf >>>>> > in 3rdparty/{include,lib} (for module developers to use them without >>>>> > doing a proper mesos installation), you need to provide the correct >>>>> > "--prefix" flag that points to 3rdparty/. However, due to multiple >>>>> > levels of configure.ac, the "--prefix" can at best be generated by >>>>> > prepending "../../../" to get to the great-grandparent directory. This >>>>> > is because we have a separate configure.ac which manages >>>>> > 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it, >>>>> > but they are not clean. >>>>> > >>>>> > Similar thing holds for system-wide installation of these 3rdparty >>>>> > packages. For example, ideally, we would want to use >>>>> > "${pkglibdir}/3rdparty" as a prefix for those packages. However, since >>>>> > they are part of libprocess package, we don't get the correct >>>>> > directory and have to use either hardwired $pkglibdir, or somehow pass >>>>> > it from the top-level configure all the way down to >>>>> > 3rdparty/libprocess/3rdparty/Makefile.am :-(. >>>>> > >>>>> > >>>>> >> The only reason you mentioned in the original email is that "in the >>>>> current >>>>> >> code base, we don't strictly follow the 3rdparty structure", which IMO >>>>> is >>>>> >> not a very convincing reason for such a big change. >>>>> > >>>>> > How about a not so big change? :-). What if we just move >>>>> > 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving >>>>> > stout as is? That is not a big change since we are not touching >>>>> > libprocess/stout. Just adjusting Makefiles and I am pretty sure it >>>>> > will be cleaner and simpler than what we have right now. >>>>> > >>>>> > As a later time, we can then consider moving stout out to 3rdparty/ >>>>> > while leaving libprocess as is. But that's something we can decide >>>>> > later and leave stout as an exception for now. >>>>> > >>>>> > BTW, if we were to install all the 3rdparty packages in 3rdparty/, >>>>> > that would also cut down a lot on the compiler flags (i.e., fewer "-I" >>>>> > and "-L" flags) :-). >>>>> > >>>>> > Kapil >>>>> > >>>>> >> >>>>> >> - Jie >>>>> >> >>>>> >> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> wrote: >>>>> >> >>>>> >>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yujie....@gmail.com> wrote: >>>>> >>>> >>>>> >>>>> >>>>> >>>>> However, in the current code base, we don't strictly follow the >>>>> >>> 3rdparty >>>>> >>>>> structure. For example, stout has a dependency on picojson and >>>>> >>>>> google-protobuf, but we don't put these two packages inside >>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/. >>>>> >>>> >>>>> >>>> >>>>> >>>> My understanding is that stout is header only. So it does not have to >>>>> >>>> bundle 3rdparty libraries. The user of stout is responsible for >>>>> bundling >>>>> >>>> them if they are used. >>>>> >>> >>>>> >>> >>>>> >>> I don't think being header-only is an excuse to have a broken >>>>> >>> installation :-). Further, we don't make it easier for the user to get >>>>> >>> the 3rdparty binaries either. For example, if the user has a different >>>>> >>> version of protobuf installed on the system, the compilation of any >>>>> >>> program that uses stout will fail spectacularly! >>>>> >>> >>>>> >>> Having said that, the gist here is that we have somewhat deviated from >>>>> >>> original motivation behind the 3rdparty directory and it would be nice >>>>> >>> if we can have a flatter structure. >>>>> >>> >>>>> >>>> >>>>> >>>> >>>>> >>>> - Jie >>>>> >>>> >>>>> >>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io> >>>>> wrote: >>>>> >>>> >>>>> >>>>> Hi All, >>>>> >>>>> >>>>> >>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into >>>>> >>> 3rdparty/. >>>>> >>>>> (Optionally) Move libprocess/stout to the top-level directory. >>>>> >>>>> >>>>> >>>>> I wanted to start some discussion around reorganizing stuff inside >>>>> >>>>> "3rdparty". I apologize for the length of the email, please bear >>>>> >>>>> with >>>>> >>> me. >>>>> >>>>> >>>>> >>>>> Traditionally, 3rdparty has been used to hold all Mesos dependencies >>>>> >>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further, >>>>> >>>>> libprocess/3rdparty was to hold all libprocess dependencies (which >>>>> may >>>>> >>> in >>>>> >>>>> turn be Mesos dependencies as well). >>>>> >>>>> >>>>> >>>>> As I understand, the original motivation was to emphasize that >>>>> >>> libprocess >>>>> >>>>> is an independent project which depends on "stout", which in turn is >>>>> >>> also >>>>> >>>>> an independent project. >>>>> >>>>> >>>>> >>>>> However, in the current code base, we don't strictly follow the >>>>> >>> 3rdparty >>>>> >>>>> structure. For example, stout has a dependency on picojson and >>>>> >>>>> google-protobuf, but we don't put these two packages inside >>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/. >>>>> >>>>> >>>>> >>>>> In light of these anomalies, I want to propose that we flatten out >>>>> the >>>>> >>>>> 3rdparty directory and put all packages (libprocess, stout, >>>>> >>>>> protobuf, >>>>> >>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We can >>>>> still >>>>> >>> use >>>>> >>>>> "--with-XYZ=..." to the full extent as needed. >>>>> >>>>> >>>>> >>>>> To take it a step further, I want to propose that we bring >>>>> >>>>> libprocess >>>>> >>> and >>>>> >>>>> stout out of 3rdparty/ and move them at the top level (i.e., make >>>>> them >>>>> >>>>> peers of src/). That way, all code in 3rdparty/ is stuff from >>>>> >>>>> "third" >>>>> >>>>> parties and is used only when "--with-bundled" is defined (by >>>>> default). >>>>> >>>>> This hierarchy will still allow us to keep libprocess and stout as >>>>> >>> separate >>>>> >>>>> independent projects. >>>>> >>>>> >>>>> >>>>> The motivation for this proposal came when dealing with 3rdparty >>>>> >>>>> dependencies for module development. A module developer needs access >>>>> to >>>>> >>>>> protobuf, picojson, glog, etc., and for that matter, the exact >>>>> >>> versions of >>>>> >>>>> these packages that Mesos was compiled with. >>>>> >>>>> >>>>> >>>>> We want to solve this problem by installing module-specific 3rdparty >>>>> >>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make install" >>>>> (if >>>>> >>>>> configured with something like >>>>> "--enable-install-module-dependencies"). >>>>> >>>>> (There is a discussion going on in a separate thread). >>>>> >>>>> >>>>> >>>>> Further, as of today, when we install Mesos using "make install", we >>>>> >>>>> install stout headers in "${prefix}/include/stout". However, stout >>>>> has >>>>> >>>>> dependencies on picojson[1] and google-protobuf headers which may >>>>> >>>>> not >>>>> >>> be >>>>> >>>>> present on the machine. This leaves stout, and in turn libprocess >>>>> >>>>> and >>>>> >>> Mesos >>>>> >>>>> headers, fairly broken. I understand that this issue is somewhat >>>>> >>> orthogonal >>>>> >>>>> to the main issue being discussed in this mail, but I wanted to put >>>>> it >>>>> >>> out >>>>> >>>>> since it's related. >>>>> >>>>> >>>>> >>>>> Any thoughts, comments, concerns are most welcome! >>>>> >>>>> >>>>> >>>>> Best, >>>>> >>>>> Kapil >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> [1]: Picojson issue was resolved as part of >>>>> >>>>> https://reviews.apache.org/r/41424/ which installs picojson.h into >>>>> the >>>>> >>>>> include-dir. >>>>> >>>>> >>>>> >>> >>>>> >>>>> >> >> >> >> -- >> ~Kevin > > > > -- > Alex > > Theory is the first term in the Taylor series of practice. -- Thomas M > Cover (1992) -- Alex Theory is the first term in the Taylor series of practice. -- Thomas M Cover (1992)
