Re: [Monotone-devel] library-build progress report and request for help with botan
On Wed, Apr 9, 2008 at 1:02 PM, Markus Schiltknecht <[EMAIL PROTECTED]> wrote: > Trying that setup, I've been puzzled about all the steps needed. I still > don't get warm with the mixture of the configuration step and the actual > building step. However, that might just be personal preference. It is unusual, but I see no alternative, given that we must be able to control the command line switches passed to subdirectory configure scripts, and AC_CONFIG_SUBDIRS has no support for this. (Also, AC_CONFIG_SUBDIRS does not know how to run botan's configure.pl correctly.) I also think that the configure script for the monotone subdirectory will be simpler if it is run after the (enabled subset of the) bundled libraries are installed to the staging directory, but as I haven't yet revised that script, I don't know for sure. > * I've spent some time to figure out that I needed to run aclocal for > every subdir. Only later on, I've noticed that there's an autogen.sh script > which does that for me. However, as INSTALL still recommends running aclocal > manually, that might entrap other users as well. I intended to edit INSTALL only after everything was done, but we can certainly change it now if you think that's more useful. (You mean autoreconf, right?) > * If pcre's configure script fails after having created pcre/Makefile (for > example due to missing config.h.in), a subsequent 'make pcre/Makefile' won't > restart configure for pcre, because the Makefile is up-to-date. Maybe simply > add more of the files there as dependencies, so that make reruns pcre's > configure. I didn't do that only because I get a headache if I try to write down the set of rules that would be necessary, and I thought I could get away with not doing it yet. > * Botan's 'make install' complains about ownership problems: Jack's fixed this upstream, but for now you can do 'make install USER=u GROUP=g' where u and g are your username and primary group, respectively. I mean to put that into the top-level Makefile until a new botan comes out. > Do we really need to run 'make install' for these 3rdparty libraries? Yes. If we instead use -I to pick up headers from the source tree, builds with the bundled libraries will be able to see library-internal headers, whereas builds with system libraries won't; this is asking for trouble. Also, running 'make install' means that the monotone subdirectory doesn't have to know anything about the structure of the bundled library subdirectories; it just does -I libinst/include -L libinst/lib and it's happy. > Having top-level 'make' recurse into all of subdirs running 'make install' > every time seems expensive to me. This is what libinst/*-stamp are for. A second 'make' at top level shouldn't recurse. Of course, we need real dependencies for those so that it does rebuild a subdirectory that's changed, but again, that is something I thought I could get away with implementing later. > After having worked around these issues, I've been able to run 'make' and > 'make install' in all of the subdirectories (lua, botan, pcre, netxx, idna > and sqlite). Cool beans. Did you get the autoconf tests that are for sqlite's benefit out of monotone/configure.ac and into the stub sqlite/configure.ac, or did you drop in the upstream sqlite build harness, or? A nice next-action would be to get botan and sqlite's internal testsuites imported and working. zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build progress report and request for help with botan
Hi, I've finally got around playing with that branch again. Some remarks below. Zack Weinberg wrote: Monotone itself has been moved to another subdirectory (currently named "monotone", but I could be persuaded to rename it "src" or something like that). +1 for naming it "src". The top-level configure script just saves all its arguments into a Makefile. That Makefile invokes the library configure scripts with --prefix set to a subdirectory "libinst" of the build tree, then builds them and installs them. It then runs the monotone-subdirectory configure script, telling it to look in "libinst" for its libraries. Trying that setup, I've been puzzled about all the steps needed. I still don't get warm with the mixture of the configuration step and the actual building step. However, that might just be personal preference. Trying to build the whole thing, I've encountered the following problems: * I've spent some time to figure out that I needed to run aclocal for every subdir. Only later on, I've noticed that there's an autogen.sh script which does that for me. However, as INSTALL still recommends running aclocal manually, that might entrap other users as well. * If pcre's configure script fails after having created pcre/Makefile (for example due to missing config.h.in), a subsequent 'make pcre/Makefile' won't restart configure for pcre, because the Makefile is up-to-date. Maybe simply add more of the files there as dependencies, so that make reruns pcre's configure. * Botan's 'make install' complains about ownership problems: install: cannot change ownership of `/home/markus/projects/monotone/build.nvm.library-build/libinst/share/doc/Botan-1.7.4/credits.txt': Operation not permitted Do we really need to run 'make install' for these 3rdparty libraries? Having top-level 'make' recurse into all of subdirs running 'make install' every time seems expensive to me. After having worked around these issues, I've been able to run 'make' and 'make install' in all of the subdirectories (lua, botan, pcre, netxx, idna and sqlite). Don't even try building the monotone subdirectory yet. People wanting to experiment may find the toplevel targets "make /Makefile" and "make libinst/-stamp" of use. Jup, even the -stamp targets are working. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build progress report and request for help with botan
On Sun, Apr 6, 2008 at 9:40 AM, Markus Schiltknecht <[EMAIL PROTECTED]> wrote: > First of all, have you pulled from my server, nabagan.bluegap.ch? AFAICT > that's the only place hosting the newest revisions of that branch. (As I > couldn't push to the original server, nor monotone.ca). Yes, I have. > If we stick to that special and troublesome stripped botan variant For the record, I only want to do that for the initial work on this branch. After it's done and merged I think an update to a full copy of botan should be very high on the priority list, and I'm good with just using tarballs. Thanks for the explanation of how to do the one-file addition. zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build progress report and request for help with botan
Hi, Zack Weinberg wrote: On Sat, Apr 5, 2008 at 12:37 PM, Zack Weinberg <[EMAIL PROTECTED]> wrote: On Sat, Apr 5, 2008 at 6:41 AM, Markus Schiltknecht <[EMAIL PROTECTED]> wrote: > I was trying to keep the current, stripped botan variant, so as to keep > changes to a minimum. That one gets propagated from ... uhm... > au...matthew.something.somewhere... see botan/README.monotone, I've updated > comments in there. I'll look at that. I *think* the fix is simple - it appears to just want one more file - but I was a little scared of the merge process last night. First of all, have you pulled from my server, nabagan.bluegap.ch? AFAICT that's the only place hosting the newest revisions of that branch. (As I couldn't push to the original server, nor monotone.ca). If we stick to that special and troublesome stripped botan variant, we should first make sure we all have access to that branch. Richard, can you organize write permissions to "au.asn.ucc.matt.botan.monotone-2" for at least me and Zack on monotone.ac? Or shall we better rename that branch to "nvm.something..."? OTOH, that would pull the whole botan repository into the monotone namespace (i.e. net.venge.monotone*) which is what Matthew was trying to avoid. Ok, I've confirmed that the major problem with the botan subdirectory is that readme.txt is missing, but I'm not 100% sure I understand how to pull in things from au.asn.ucc.matt.botan.monotone-2 ... the file is in the Attic there. Before I do anything I would appreciate it if someone could confirm that this is the right approach. The Attic is to avoid die die die merge problems. Just move readme.txt from the Attic back to it's original location. That way, propagating from net.randombit.botan keeps working as expected. 1) in aaumb, mtn mv Attic/readme.txt readme.txt ; commit. 2) copy the file from the aaumb checkout to a nvm.botan checkout's botan subdirectory, "mtn add" it, and add it to EXTRA_DIST in the top-level makefile. Commit. 3) Propagate from nvm.botan to nvm. 4) Propagate from nvm to nvm.library-build. (I think the file will just magically appear, because the "botan" subdirectory in nvm.library-build has the same inode as the one in nvm... am I right?) That looks correct. You could even skip propagating to nvm and instead propagate from nvm.botan to nvm.library-build directly. This updating process is rather expensive in terms of work needed for an upgrade of the provided botan version. I was hoping that we can replace that process with simply dropping a tarball of the newest botan library into our botan/ subdirectory and automate the rest. Of course, if the build process (configure.pl and make) would change, we'd still have to adapt. But that's much less work than having to manually add, replace or delete files around, as I had to do in the past. So, opposed to the other subprojects, I'd now vote for removing our stripped botan variant, including that staging branch (the aaumb one) and land a (full) botan 1.7.4 there, for the sake of simplifying later upgrades. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build progress report and request for help with botan
On Fri, Apr 04, 2008, Zack Weinberg wrote: > nvm.library-build is my project to build all our bundled libraries in > a way that more closely resembles how they're built upstream. (This > will also facilitate optional use of system-provided versions.) It's > intended to go like this: Each bundled library has its own > subdirectory, just as they do now, but those subdirs have Makefiles > and configure scripts of their own. Monotone itself has been moved to > another subdirectory (currently named "monotone", but I could be > persuaded to rename it "src" or something like that). The top-level > configure script just saves all its arguments into a Makefile. That > Makefile invokes the library configure scripts with --prefix set to a > subdirectory "libinst" of the build tree, then builds them and > installs them. It then runs the monotone-subdirectory configure > script, telling it to look in "libinst" for its libraries. I'm not convinced whether the "install" step here is really necessary here. About a year ago, for RPM 5 I've implemented a rather sophisticated reusable Autoconf macro RPM_CHECK_LIB (see attachment) which allows us to build against the dozen third-party libraries RPM requires in both internal/embedded and external variant. We put really lots of efforts into this scheme in 2007 and it showed up that it works very good in practice. And we do not require the library to be really installed here (as this usually increases the disk space requirements during build-time even more). Perhaps this approach could be also useful for Monotone here. > Right now what works is building idna, lua, netxx, and pcre. My > original intent was to have these be verbatim upstream tarballs but > that proved impractical, I've pretty much written my own Makefiles and > configure.ac's for all of them. [Only idna and pcre have upstream > build harnesses that are any use, and they both have tons of unrelated > junk that we don't want cluttering up the repository.] I think if you don't have to install the libraries you usually can use the Makefile's out-of-the-box and really keep the stuff fully verbatim. > botan builds but does not install, because we are using the upstream > build harness for that, and it wants files that were left out of the > original import. I would appreciate help from people who've done > upstream botan imports on that front. sqlite doesn't build - I just > haven't gotten to that one yet. I'd like to know, though, whether we > want to move to sqlite 3.5.7 separately or as part of this project > (looking at the big picture, my inclination is to say "separately", > but doing it as part of this project might make this project a wee > easier - you'll note that I updated the idna directory, where the > tradeoff is clearer). Current status of my evaluation "SQLite 3.5.x for Monotone" is: | Current branch: net.venge.monotone | Changes against parent 36b6cb016019e675bc24a2fa97d4eb19fa4919af | dropped sqlite | dropped sqlite/alter.c | dropped sqlite/analyze.c | dropped sqlite/attach.c | dropped sqlite/auth.c | dropped sqlite/btree.c | dropped sqlite/btree.h | dropped sqlite/btreeInt.h | dropped sqlite/build.c | dropped sqlite/callback.c | dropped sqlite/complete.c | dropped sqlite/date.c | dropped sqlite/delete.c | dropped sqlite/expr.c | dropped sqlite/func.c | dropped sqlite/hash.c | dropped sqlite/hash.h | dropped sqlite/insert.c | dropped sqlite/keywordhash.h | dropped sqlite/legacy.c | dropped sqlite/loadext.c | dropped sqlite/main.c | dropped sqlite/malloc.c | dropped sqlite/opcodes.c | dropped sqlite/opcodes.h | dropped sqlite/os.c | dropped sqlite/os.h | dropped sqlite/os_common.h | dropped sqlite/os_os2.c | dropped sqlite/os_os2.h | dropped sqlite/os_unix.c | dropped sqlite/os_win.c | dropped sqlite/pager.c | dropped sqlite/pager.h | dropped sqlite/parse.c | dropped sqlite/parse.h | dropped sqlite/pragma.c | dropped sqlite/prepare.c | dropped sqlite/printf.c | dropped sqlite/random.c | dropped sqlite/select.c | dropped sqlite/sqlite3.h | dropped sqlite/sqlite3ext.h | dropped sqlite/sqliteInt.h | dropped sqlite/sqliteLimit.h | dropped sqlite/table.c | dropped sqlite/tokenize.c | dropped sqlite/trigger.c | dropped sqlite/update.c | dropped sqlite/utf.c | dropped sqlite/util.c | dropped sqlite/vacuum.c | dropped sqlite/vdbe.c | dropped sqlite/vdbe.h | dropped sqlite/vdbeInt.h | dropped sqlite/vdbeapi.c | dropped sqlite/vdbeaux.c | dropped sqlite/vdbeblob.c | dropped sqlite/vdbefifo.c | dropped sqlite/vdbemem.c | dropped sqlite/vtab.c | dropped sqlite/where.c | addedsqlite3.c | addedsqlite3.h | patched Makefile.am | patched database.cc | patched m4/sqlite.m4 | patched schema_migration.cc | patched tests/fail_cleanly_on_unreadable_db/__driver__.lua As you see, I've replaced the s
Re: [Monotone-devel] library-build progress report and request for help with botan
On Sat, Apr 5, 2008 at 12:37 PM, Zack Weinberg <[EMAIL PROTECTED]> wrote: > On Sat, Apr 5, 2008 at 6:41 AM, Markus Schiltknecht <[EMAIL PROTECTED]> wrote: > > I was trying to keep the current, stripped botan variant, so as to keep > > changes to a minimum. That one gets propagated from ... uhm... > > au...matthew.something.somewhere... see botan/README.monotone, I've updated > > comments in there. > > I'll look at that. I *think* the fix is simple - it appears to just > want one more file - but I was a little scared of the merge process > last night. Ok, I've confirmed that the major problem with the botan subdirectory is that readme.txt is missing, but I'm not 100% sure I understand how to pull in things from au.asn.ucc.matt.botan.monotone-2 ... the file is in the Attic there. Before I do anything I would appreciate it if someone could confirm that this is the right approach. 1) in aaumb, mtn mv Attic/readme.txt readme.txt ; commit. 2) copy the file from the aaumb checkout to a nvm.botan checkout's botan subdirectory, "mtn add" it, and add it to EXTRA_DIST in the top-level makefile. Commit. 3) Propagate from nvm.botan to nvm. 4) Propagate from nvm to nvm.library-build. (I think the file will just magically appear, because the "botan" subdirectory in nvm.library-build has the same inode as the one in nvm... am I right?) zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build progress report and request for help with botan
On Sat, Apr 5, 2008 at 6:41 AM, Markus Schiltknecht <[EMAIL PROTECTED]> wrote: > Well, if cluttering up the repository is the only counter argument, I'd say > better use their build harnesses, because maintenance and upgrading becomes > much simpler. Otherwise, we will have to maintain our own build harness for > that, which is about as bad as it is today. > > But it sounds like you hit other problems as well. I'm just pointing out > the maintenance aspect again... There's a bit more to it than that; we'd wind up having to modify their build harnesses anyway for things like ensuring that we only have one copy of config.guess/config.sub in the tree, idna has pieces that are GPLv3, autopoint was not cooperating with me when run in a subdirectory, etc. etc. etc. I rather *want* to have each subdirectory be just as it was in the original, but right now I don't think that's going to work. > I was trying to keep the current, stripped botan variant, so as to keep > changes to a minimum. That one gets propagated from ... uhm... > au...matthew.something.somewhere... see botan/README.monotone, I've updated > comments in there. I'll look at that. I *think* the fix is simple - it appears to just want one more file - but I was a little scared of the merge process last night. > It's the same 'cluttering' vs 'simple maintenance' decision: we currently > have a sleek and stripped botan for monotone. I didn't dare giving that up > for simpler maintenance, yet. But in the long run, we *should* replace it > with the tarball, I think. Cluttering the repository is much less expensive > for us, than having to clean up our own test harness for every botan > release. I definitely think we want to go that way in the long run, but I'd rather do that as a separate change than this. > I'd certainly vote for decoupling these changes, because it might actually > be *less* work in the end, due to simpler testing and debugging. Ok. > > People wanting to experiment may find the toplevel targets "make > > /Makefile" and "make libinst/-stamp" of use. > > This sounds like you've displaced autoconf somewhat. I've thought it would > be clearer if ./configure would also configure the sub directories. What > were the reasons for doing that from make? The subdirectory configures must be run from make so we can mess with their arguments. There is no way to do that from the top-level configure script. Also, this lets me configure the monotone subdirectory *after* all the libraries are built and installed -- then it doesn't have to know anything about user selection of bundled versus system libraries. zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build progress report and request for help with botan
Hi, Zack Weinberg wrote: Right now what works is building idna, lua, netxx, and pcre. My original intent was to have these be verbatim upstream tarballs but that proved impractical, I've pretty much written my own Makefiles and configure.ac's for all of them. [Only idna and pcre have upstream build harnesses that are any use, and they both have tons of unrelated junk that we don't want cluttering up the repository.] Well, if cluttering up the repository is the only counter argument, I'd say better use their build harnesses, because maintenance and upgrading becomes much simpler. Otherwise, we will have to maintain our own build harness for that, which is about as bad as it is today. But it sounds like you hit other problems as well. I'm just pointing out the maintenance aspect again... botan builds but does not install, because we are using the upstream build harness for that, and it wants files that were left out of the original import. I would appreciate help from people who've done upstream botan imports on that front. I was trying to keep the current, stripped botan variant, so as to keep changes to a minimum. That one gets propagated from ... uhm... au...matthew.something.somewhere... see botan/README.monotone, I've updated comments in there. But maybe it's not worth maintaining that upgrading strategy. Instead, let's just replace it with the original tarball. It's the same 'cluttering' vs 'simple maintenance' decision: we currently have a sleek and stripped botan for monotone. I didn't dare giving that up for simpler maintenance, yet. But in the long run, we *should* replace it with the tarball, I think. Cluttering the repository is much less expensive for us, than having to clean up our own test harness for every botan release. sqlite doesn't build - I just haven't gotten to that one yet. I'd like to know, though, whether we want to move to sqlite 3.5.7 separately or as part of this project (looking at the big picture, my inclination is to say "separately", but doing it as part of this project might make this project a wee easier - you'll note that I updated the idna directory, where the tradeoff is clearer). I'd certainly vote for decoupling these changes, because it might actually be *less* work in the end, due to simpler testing and debugging. Don't even try building the monotone subdirectory yet. People wanting to experiment may find the toplevel targets "make /Makefile" and "make libinst/-stamp" of use. This sounds like you've displaced autoconf somewhat. I've thought it would be clearer if ./configure would also configure the sub directories. What were the reasons for doing that from make? Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] library-build progress report and request for help with botan
nvm.library-build is my project to build all our bundled libraries in a way that more closely resembles how they're built upstream. (This will also facilitate optional use of system-provided versions.) It's intended to go like this: Each bundled library has its own subdirectory, just as they do now, but those subdirs have Makefiles and configure scripts of their own. Monotone itself has been moved to another subdirectory (currently named "monotone", but I could be persuaded to rename it "src" or something like that). The top-level configure script just saves all its arguments into a Makefile. That Makefile invokes the library configure scripts with --prefix set to a subdirectory "libinst" of the build tree, then builds them and installs them. It then runs the monotone-subdirectory configure script, telling it to look in "libinst" for its libraries. Right now what works is building idna, lua, netxx, and pcre. My original intent was to have these be verbatim upstream tarballs but that proved impractical, I've pretty much written my own Makefiles and configure.ac's for all of them. [Only idna and pcre have upstream build harnesses that are any use, and they both have tons of unrelated junk that we don't want cluttering up the repository.] botan builds but does not install, because we are using the upstream build harness for that, and it wants files that were left out of the original import. I would appreciate help from people who've done upstream botan imports on that front. sqlite doesn't build - I just haven't gotten to that one yet. I'd like to know, though, whether we want to move to sqlite 3.5.7 separately or as part of this project (looking at the big picture, my inclination is to say "separately", but doing it as part of this project might make this project a wee easier - you'll note that I updated the idna directory, where the tradeoff is clearer). Don't even try building the monotone subdirectory yet. People wanting to experiment may find the toplevel targets "make /Makefile" and "make libinst/-stamp" of use. zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
