Thanks for bringing it up Alexander! I don't have a strong opinion wrt git submodules since I don't have much experience with them personally. Having said that, I would like to go conservative on this one (baby steps :-) ).
Further, I do understand that moving libprocess and stout directories will be painful for people who already have several branches and will have conflicts. But I do think, there are some interim solutions as well (for example, move libprocess/stout to wherever we want, but keep a symlink from 3rdparty/libprocess, etc, to those new locations for some time). I am sure there are better solutions out there, but this should work too. Best, Kapil On Tue, Feb 16, 2016 at 12:38 PM, Erik Weathers <eweath...@groupon.com.invalid> wrote: > If we go to git submodules, please ensure there are good docs around how to > update cloned repos. > > e.g., From ansible: https://docs.ansible.com/ansible/intro_installation.html > > Note when updating ansible, be sure to not only update the source tree, but > also the “submodules” in git which point at Ansible’s own modules (not the > same kind of modules, alas). > > $ git pull --rebase > $ git submodule update --init --recursive > > Thanks, > > - Erik > > On Tue, Feb 16, 2016 at 8:54 AM, Alexander Rojas <alexan...@mesosphere.io> > wrote: > >> +1 >> I am one who is totally in for that change. It is not only the directories >> problem, but the structure which has led that the stout tests (which do >> need to be compiled) are actually managed in the libprocess Makefile, on >> top of all the things you have already mentioned. >> >> >> > On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io> wrote: >> > >> > On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yujie....@gmail.com> wrote: >> >> Kapil, >> >> >> >> I guess what I want to understand is why the existing structure makes it >> >> hard for you to do the things that you want to do (installing >> >> module-specific 3rdparty dependencies into "${pkglibdir}/3rdparty" as >> part >> >> of "make install"). >> > >> > Let me see if I can answer that :-). >> > >> > This is somewhat related. For example, if we want to install protobuf >> > in 3rdparty/{include,lib} (for module developers to use them without >> > doing a proper mesos installation), you need to provide the correct >> > "--prefix" flag that points to 3rdparty/. However, due to multiple >> > levels of configure.ac, the "--prefix" can at best be generated by >> > prepending "../../../" to get to the great-grandparent directory. This >> > is because we have a separate configure.ac which manages >> > 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it, >> > but they are not clean. >> > >> > Similar thing holds for system-wide installation of these 3rdparty >> > packages. For example, ideally, we would want to use >> > "${pkglibdir}/3rdparty" as a prefix for those packages. However, since >> > they are part of libprocess package, we don't get the correct >> > directory and have to use either hardwired $pkglibdir, or somehow pass >> > it from the top-level configure all the way down to >> > 3rdparty/libprocess/3rdparty/Makefile.am :-(. >> > >> > >> >> The only reason you mentioned in the original email is that "in the >> current >> >> code base, we don't strictly follow the 3rdparty structure", which IMO >> is >> >> not a very convincing reason for such a big change. >> > >> > How about a not so big change? :-). What if we just move >> > 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving >> > stout as is? That is not a big change since we are not touching >> > libprocess/stout. Just adjusting Makefiles and I am pretty sure it >> > will be cleaner and simpler than what we have right now. >> > >> > As a later time, we can then consider moving stout out to 3rdparty/ >> > while leaving libprocess as is. But that's something we can decide >> > later and leave stout as an exception for now. >> > >> > BTW, if we were to install all the 3rdparty packages in 3rdparty/, >> > that would also cut down a lot on the compiler flags (i.e., fewer "-I" >> > and "-L" flags) :-). >> > >> > Kapil >> > >> >> >> >> - Jie >> >> >> >> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> wrote: >> >> >> >>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yujie....@gmail.com> wrote: >> >>>> >> >>>>> >> >>>>> However, in the current code base, we don't strictly follow the >> >>> 3rdparty >> >>>>> structure. For example, stout has a dependency on picojson and >> >>>>> google-protobuf, but we don't put these two packages inside >> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/. >> >>>> >> >>>> >> >>>> My understanding is that stout is header only. So it does not have to >> >>>> bundle 3rdparty libraries. The user of stout is responsible for >> bundling >> >>>> them if they are used. >> >>> >> >>> >> >>> I don't think being header-only is an excuse to have a broken >> >>> installation :-). Further, we don't make it easier for the user to get >> >>> the 3rdparty binaries either. For example, if the user has a different >> >>> version of protobuf installed on the system, the compilation of any >> >>> program that uses stout will fail spectacularly! >> >>> >> >>> Having said that, the gist here is that we have somewhat deviated from >> >>> original motivation behind the 3rdparty directory and it would be nice >> >>> if we can have a flatter structure. >> >>> >> >>>> >> >>>> >> >>>> - Jie >> >>>> >> >>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io> >> wrote: >> >>>> >> >>>>> Hi All, >> >>>>> >> >>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into >> >>> 3rdparty/. >> >>>>> (Optionally) Move libprocess/stout to the top-level directory. >> >>>>> >> >>>>> I wanted to start some discussion around reorganizing stuff inside >> >>>>> "3rdparty". I apologize for the length of the email, please bear with >> >>> me. >> >>>>> >> >>>>> Traditionally, 3rdparty has been used to hold all Mesos dependencies >> >>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further, >> >>>>> libprocess/3rdparty was to hold all libprocess dependencies (which >> may >> >>> in >> >>>>> turn be Mesos dependencies as well). >> >>>>> >> >>>>> As I understand, the original motivation was to emphasize that >> >>> libprocess >> >>>>> is an independent project which depends on "stout", which in turn is >> >>> also >> >>>>> an independent project. >> >>>>> >> >>>>> However, in the current code base, we don't strictly follow the >> >>> 3rdparty >> >>>>> structure. For example, stout has a dependency on picojson and >> >>>>> google-protobuf, but we don't put these two packages inside >> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/. >> >>>>> >> >>>>> In light of these anomalies, I want to propose that we flatten out >> the >> >>>>> 3rdparty directory and put all packages (libprocess, stout, protobuf, >> >>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We can >> still >> >>> use >> >>>>> "--with-XYZ=..." to the full extent as needed. >> >>>>> >> >>>>> To take it a step further, I want to propose that we bring libprocess >> >>> and >> >>>>> stout out of 3rdparty/ and move them at the top level (i.e., make >> them >> >>>>> peers of src/). That way, all code in 3rdparty/ is stuff from "third" >> >>>>> parties and is used only when "--with-bundled" is defined (by >> default). >> >>>>> This hierarchy will still allow us to keep libprocess and stout as >> >>> separate >> >>>>> independent projects. >> >>>>> >> >>>>> The motivation for this proposal came when dealing with 3rdparty >> >>>>> dependencies for module development. A module developer needs access >> to >> >>>>> protobuf, picojson, glog, etc., and for that matter, the exact >> >>> versions of >> >>>>> these packages that Mesos was compiled with. >> >>>>> >> >>>>> We want to solve this problem by installing module-specific 3rdparty >> >>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make install" >> (if >> >>>>> configured with something like >> "--enable-install-module-dependencies"). >> >>>>> (There is a discussion going on in a separate thread). >> >>>>> >> >>>>> Further, as of today, when we install Mesos using "make install", we >> >>>>> install stout headers in "${prefix}/include/stout". However, stout >> has >> >>>>> dependencies on picojson[1] and google-protobuf headers which may not >> >>> be >> >>>>> present on the machine. This leaves stout, and in turn libprocess and >> >>> Mesos >> >>>>> headers, fairly broken. I understand that this issue is somewhat >> >>> orthogonal >> >>>>> to the main issue being discussed in this mail, but I wanted to put >> it >> >>> out >> >>>>> since it's related. >> >>>>> >> >>>>> Any thoughts, comments, concerns are most welcome! >> >>>>> >> >>>>> Best, >> >>>>> Kapil >> >>>>> >> >>>>> >> >>>>> [1]: Picojson issue was resolved as part of >> >>>>> https://reviews.apache.org/r/41424/ which installs picojson.h into >> the >> >>>>> include-dir. >> >>>>> >> >>> >> >>
