Folks: Took about 30 minutes to put together a prototype of the great `3rdparty` flattening. Please see the (extremely preliminary) review[1] or my working branch[2] for a really close approximation of the number of changes to the CMake build system we'd need to support this flattening.
Overall, the changes are extremely small. We (apparently) did a pretty good job of making the CMake configuration scripts modular and consumable by arbitrary projects, so the changes are mostly things like "move the config `include` call over there instead of being over there" and "change a handful of path variables to reflect the new dir structure". (I hope, btw, that it doesn't seem arrogant to say this, but I think it is justified given the relatively minor changes in the actual CMake code.) Feel free to remix/take/throw away any subset of this. I don't need any credit for it in the final patch if someone marches off in that direction. :) [1] https://reviews.apache.org/r/44099/ [2] https://github.com/hausdorff/mesos/commits/prototype_flattened_3rd On Thu, Feb 18, 2016 at 12:22 PM, Kevin Klues <klue...@gmail.com> wrote: > I am also a fan of git submodules in the long term, but avoiding them > in the short term. We should get things organized as we want them > first, and then start thinking about pulling libprocess/stout out into > submodules later (while also preserving their history!) > > I disagree with moving libprocess and stout up to the same level as > src/. If we want to make sure they don't bleed into Mesos proper, we > really should treat them the same as any other 3rdparty code that we > depend on. This will become more relevant when/if we move them into > submodules. > > Given all that, the only real challenge with flattening our 3rdparty > dependencies into a single folder should be the changes we have to > make to our configure.ac and Makefile.am scripts to know where to look > for their dependencies now. In the end these should be URLs to > versioned tarballs that we host somewhere (or git repos that we can > have forked and tagged with specific versions). In the short term > these can just be relative paths in the mesos tree though. > > On Tue, Feb 16, 2016 at 1:26 PM, Kapil Arya <ka...@mesosphere.io> wrote: >> Thanks for bringing it up Alexander! >> >> I don't have a strong opinion wrt git submodules since I don't have >> much experience with them personally. Having said that, I would like >> to go conservative on this one (baby steps :-) ). >> >> Further, I do understand that moving libprocess and stout directories >> will be painful for people who already have several branches and will >> have conflicts. But I do think, there are some interim solutions as >> well (for example, move libprocess/stout to wherever we want, but keep >> a symlink from 3rdparty/libprocess, etc, to those new locations for >> some time). I am sure there are better solutions out there, but this >> should work too. >> >> Best, >> Kapil >> >> On Tue, Feb 16, 2016 at 12:38 PM, Erik Weathers >> <eweath...@groupon.com.invalid> wrote: >>> If we go to git submodules, please ensure there are good docs around how to >>> update cloned repos. >>> >>> e.g., From ansible: https://docs.ansible.com/ansible/intro_installation.html >>> >>> Note when updating ansible, be sure to not only update the source tree, but >>> also the “submodules” in git which point at Ansible’s own modules (not the >>> same kind of modules, alas). >>> >>> $ git pull --rebase >>> $ git submodule update --init --recursive >>> >>> Thanks, >>> >>> - Erik >>> >>> On Tue, Feb 16, 2016 at 8:54 AM, Alexander Rojas <alexan...@mesosphere.io> >>> wrote: >>> >>>> +1 >>>> I am one who is totally in for that change. It is not only the directories >>>> problem, but the structure which has led that the stout tests (which do >>>> need to be compiled) are actually managed in the libprocess Makefile, on >>>> top of all the things you have already mentioned. >>>> >>>> >>>> > On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io> wrote: >>>> > >>>> > On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yujie....@gmail.com> wrote: >>>> >> Kapil, >>>> >> >>>> >> I guess what I want to understand is why the existing structure makes it >>>> >> hard for you to do the things that you want to do (installing >>>> >> module-specific 3rdparty dependencies into "${pkglibdir}/3rdparty" as >>>> part >>>> >> of "make install"). >>>> > >>>> > Let me see if I can answer that :-). >>>> > >>>> > This is somewhat related. For example, if we want to install protobuf >>>> > in 3rdparty/{include,lib} (for module developers to use them without >>>> > doing a proper mesos installation), you need to provide the correct >>>> > "--prefix" flag that points to 3rdparty/. However, due to multiple >>>> > levels of configure.ac, the "--prefix" can at best be generated by >>>> > prepending "../../../" to get to the great-grandparent directory. This >>>> > is because we have a separate configure.ac which manages >>>> > 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it, >>>> > but they are not clean. >>>> > >>>> > Similar thing holds for system-wide installation of these 3rdparty >>>> > packages. For example, ideally, we would want to use >>>> > "${pkglibdir}/3rdparty" as a prefix for those packages. However, since >>>> > they are part of libprocess package, we don't get the correct >>>> > directory and have to use either hardwired $pkglibdir, or somehow pass >>>> > it from the top-level configure all the way down to >>>> > 3rdparty/libprocess/3rdparty/Makefile.am :-(. >>>> > >>>> > >>>> >> The only reason you mentioned in the original email is that "in the >>>> current >>>> >> code base, we don't strictly follow the 3rdparty structure", which IMO >>>> is >>>> >> not a very convincing reason for such a big change. >>>> > >>>> > How about a not so big change? :-). What if we just move >>>> > 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving >>>> > stout as is? That is not a big change since we are not touching >>>> > libprocess/stout. Just adjusting Makefiles and I am pretty sure it >>>> > will be cleaner and simpler than what we have right now. >>>> > >>>> > As a later time, we can then consider moving stout out to 3rdparty/ >>>> > while leaving libprocess as is. But that's something we can decide >>>> > later and leave stout as an exception for now. >>>> > >>>> > BTW, if we were to install all the 3rdparty packages in 3rdparty/, >>>> > that would also cut down a lot on the compiler flags (i.e., fewer "-I" >>>> > and "-L" flags) :-). >>>> > >>>> > Kapil >>>> > >>>> >> >>>> >> - Jie >>>> >> >>>> >> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> wrote: >>>> >> >>>> >>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yujie....@gmail.com> wrote: >>>> >>>> >>>> >>>>> >>>> >>>>> However, in the current code base, we don't strictly follow the >>>> >>> 3rdparty >>>> >>>>> structure. For example, stout has a dependency on picojson and >>>> >>>>> google-protobuf, but we don't put these two packages inside >>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/. >>>> >>>> >>>> >>>> >>>> >>>> My understanding is that stout is header only. So it does not have to >>>> >>>> bundle 3rdparty libraries. The user of stout is responsible for >>>> bundling >>>> >>>> them if they are used. >>>> >>> >>>> >>> >>>> >>> I don't think being header-only is an excuse to have a broken >>>> >>> installation :-). Further, we don't make it easier for the user to get >>>> >>> the 3rdparty binaries either. For example, if the user has a different >>>> >>> version of protobuf installed on the system, the compilation of any >>>> >>> program that uses stout will fail spectacularly! >>>> >>> >>>> >>> Having said that, the gist here is that we have somewhat deviated from >>>> >>> original motivation behind the 3rdparty directory and it would be nice >>>> >>> if we can have a flatter structure. >>>> >>> >>>> >>>> >>>> >>>> >>>> >>>> - Jie >>>> >>>> >>>> >>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io> >>>> wrote: >>>> >>>> >>>> >>>>> Hi All, >>>> >>>>> >>>> >>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into >>>> >>> 3rdparty/. >>>> >>>>> (Optionally) Move libprocess/stout to the top-level directory. >>>> >>>>> >>>> >>>>> I wanted to start some discussion around reorganizing stuff inside >>>> >>>>> "3rdparty". I apologize for the length of the email, please bear with >>>> >>> me. >>>> >>>>> >>>> >>>>> Traditionally, 3rdparty has been used to hold all Mesos dependencies >>>> >>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further, >>>> >>>>> libprocess/3rdparty was to hold all libprocess dependencies (which >>>> may >>>> >>> in >>>> >>>>> turn be Mesos dependencies as well). >>>> >>>>> >>>> >>>>> As I understand, the original motivation was to emphasize that >>>> >>> libprocess >>>> >>>>> is an independent project which depends on "stout", which in turn is >>>> >>> also >>>> >>>>> an independent project. >>>> >>>>> >>>> >>>>> However, in the current code base, we don't strictly follow the >>>> >>> 3rdparty >>>> >>>>> structure. For example, stout has a dependency on picojson and >>>> >>>>> google-protobuf, but we don't put these two packages inside >>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/. >>>> >>>>> >>>> >>>>> In light of these anomalies, I want to propose that we flatten out >>>> the >>>> >>>>> 3rdparty directory and put all packages (libprocess, stout, protobuf, >>>> >>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We can >>>> still >>>> >>> use >>>> >>>>> "--with-XYZ=..." to the full extent as needed. >>>> >>>>> >>>> >>>>> To take it a step further, I want to propose that we bring libprocess >>>> >>> and >>>> >>>>> stout out of 3rdparty/ and move them at the top level (i.e., make >>>> them >>>> >>>>> peers of src/). That way, all code in 3rdparty/ is stuff from "third" >>>> >>>>> parties and is used only when "--with-bundled" is defined (by >>>> default). >>>> >>>>> This hierarchy will still allow us to keep libprocess and stout as >>>> >>> separate >>>> >>>>> independent projects. >>>> >>>>> >>>> >>>>> The motivation for this proposal came when dealing with 3rdparty >>>> >>>>> dependencies for module development. A module developer needs access >>>> to >>>> >>>>> protobuf, picojson, glog, etc., and for that matter, the exact >>>> >>> versions of >>>> >>>>> these packages that Mesos was compiled with. >>>> >>>>> >>>> >>>>> We want to solve this problem by installing module-specific 3rdparty >>>> >>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make install" >>>> (if >>>> >>>>> configured with something like >>>> "--enable-install-module-dependencies"). >>>> >>>>> (There is a discussion going on in a separate thread). >>>> >>>>> >>>> >>>>> Further, as of today, when we install Mesos using "make install", we >>>> >>>>> install stout headers in "${prefix}/include/stout". However, stout >>>> has >>>> >>>>> dependencies on picojson[1] and google-protobuf headers which may not >>>> >>> be >>>> >>>>> present on the machine. This leaves stout, and in turn libprocess and >>>> >>> Mesos >>>> >>>>> headers, fairly broken. I understand that this issue is somewhat >>>> >>> orthogonal >>>> >>>>> to the main issue being discussed in this mail, but I wanted to put >>>> it >>>> >>> out >>>> >>>>> since it's related. >>>> >>>>> >>>> >>>>> Any thoughts, comments, concerns are most welcome! >>>> >>>>> >>>> >>>>> Best, >>>> >>>>> Kapil >>>> >>>>> >>>> >>>>> >>>> >>>>> [1]: Picojson issue was resolved as part of >>>> >>>>> https://reviews.apache.org/r/41424/ which installs picojson.h into >>>> the >>>> >>>>> include-dir. >>>> >>>>> >>>> >>> >>>> >>>> > > > > -- > ~Kevin -- Alex Theory is the first term in the Taylor series of practice. -- Thomas M Cover (1992)
