Re: [Monotone-devel] library build
2008/10/2 Jack Lloyd [EMAIL PROTECTED]: On Wed, Oct 01, 2008 at 07:34:20PM +0200, Markus Wanner wrote: After just having upgraded (and now landed) monotone's included botan to 1.7.12. Jack is already approaching 1.7.15 with yet another set of renaming. I'm rather going to use the nvm.stripped branch than continue to manually upgrade again. So if Botan 1.7.x is going to distros (AFAIK the only distro it is currently in is Debian) I would just as soon it be the future 1.7.15 rather than .14 since there are another slew of changes (in... BTW, 1.7.15 has minor change of API that makes some things ugly in monotone if we want to use system provided libraries. In lookup.h there are few get_cipher functions than in 1.7.14 and earlier versions where independent of Library_State and in 1.7.15 they require LibraryState as first parameter. *** It's not big deal for monotone to change to this new interface but _if_ we want use bundled system libraries that for some reason may be older than 1.7.15 we would need some #ifdefs and wrappers in monotone code. I don't question your design decisions but LibraryState is currently a singleton obtained via (AFAIR) Botan::get_default() so i don't see any reason for removing Botan::get_cipher overloads without LibraryState (correct me if I'm wrong). Old overloads can be marked as deprecated but it is wise to leave them for a while because current code base uses them. If it's part of bigger design, ,than we'll just write some wrapper functions with #ifdefs. *** New get_cipher: BOTAN_DLL Keyed_Filter* get_cipher(Library_State, const std::string, ...); Old get_cipher: BOTAN_DLL Keyed_Filter* get_cipher(const std::string, ...); Best regards, -- Zbigniew Zagórski / software developer / geek / happy daddy / ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Botan-devel] [Monotone-devel] library build
On Thu, Oct 02, 2008 at 12:45:17PM +0200, Zbigniew Zag??rski wrote: BTW, 1.7.15 has minor change of API that makes some things ugly in monotone if we want to use system provided libraries. In lookup.h there are few get_cipher functions than in 1.7.14 and earlier versions where independent of Library_State and in 1.7.15 they require LibraryState as first parameter. *** That was an initial commit cleaning up something I thought I was going to be pursuing immediately, but am probably not. That interface as it is (taking a Library_State) was purely provisional for me while I was working on other things. Eventually the entire get_cipher/lookup.h stuff is going to change I think dramatically (hopefully for the better!), but it looks like now is not the time. It's not big deal for monotone to change to this new interface but _if_ we want use bundled system libraries that for some reason may be older than 1.7.15 we would need some #ifdefs and wrappers in monotone code. This seems worthwhile. In partiuclar Debian is already shipping Botan 1.7.8 with testing (and something later, I think 1.7.12, with unstable), and Markus indicated that Monotone was compiling with 1.7.8 onwards. As I mentioned, I did not expect get_cipher taking a Library_State to be a final or stable API (in fact explicitly I plan to replace much of that code - someday), and it doesn't make sense to force Monotone to deal with it. I will revert this change for 1.7.15. This will allow Monotone to work across 1.7.8 - 1.7.15. And hopefully later on I can revisit the whole issue more in depth, probably along a branch at first. -Jack ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
Hi, Zack Weinberg wrote: It might make sense to start a new project to rip out our bundled libraries altogether and then fix everything necessary to make it work with system libraries. I've started such an effort in nvm.stripped. Beginning with: * botan (because I know that from earlier upgrades) * pcre (because it was so easy) * lua (becasue I'm been afraid of the problems you mentioned) The m4 scripts for botan and lua are stolen from the library-build build branch and adjusted. I'm not an m4 hacker, so if some m4 guru could check those that'd be great. The branch builds on my debian lenny against botan 1.7.8 (up to head), pcre 7.6 and lua 5.1 as external libraries. It passes all tests of the testsuite and weights 30 MiB (vs 37 MiB on mainline and Intel 32bit). It also became clear that there are still quite a few hacks that we rely on in our local copies of the libraries. The big one was, we compile lua as C++ so its error handling turns into C++ exceptions. A system-provided liblua won't do this. That certainly needs to be investigated. I'm currently using lua.hpp, thus including the system provided lua library as C code, yes. We don't do error propagation across the lua/C++ interface correctly anyway (errors in lua hooks have a bad way of just disappearing), so I would *like* to rip that out and go back to lua's C interface Feel free... :-) I didn't do sha1 benchmarks, but those should now use the assembler optimized variants. After just having upgraded (and now landed) monotone's included botan to 1.7.12. Jack is already approaching 1.7.15 with yet another set of renaming. I'm rather going to use the nvm.stripped branch than continue to manually upgrade again. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
On Wed, Oct 1, 2008 at 11:34 AM, Markus Wanner [EMAIL PROTECTED] wrote: I've started such an effort in nvm.stripped. Beginning with: * botan (because I know that from earlier upgrades) * pcre (because it was so easy) * lua (becasue I'm been afraid of the problems you mentioned) That sounds great, I'll give it a go on my system. The m4 scripts for botan and lua are stolen from the library-build build branch and adjusted. I'm not an m4 hacker, so if some m4 guru could check those that'd be great. Me neither, but I'll have a look and see if I notice anything. I didn't do sha1 benchmarks, but those should now use the assembler optimized variants. I'm curious to see how these go. I've been rattling emails off of Jack trying to get both the sha1_ia32 and sha1_sse2 modules to properly configure on my pentium-m and I think that's now working so I'll see how they perform here. Initial tests with botan seem to show that ia32 asm is faster than sse2 on pentium-m which seemed a bit odd. After just having upgraded (and now landed) monotone's included botan to 1.7.12. Jack is already approaching 1.7.15 with yet another set of renaming. I'm rather going to use the nvm.stripped branch than continue to manually upgrade again. I noticed that gentoo only has 1.6.5 or something so I'll see if I can encourage someone in portage land to get the botan dev stuff in as unstable. Cheers, Derek ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
Hi, Zbigniew Zagórski wrote: From my recent tests it appears that: - lua - pcre - botan - sqlite3 build without any problem on mingw/msys with simple ./configure --prefix=/mingw ; make ; make install Cool, good to know. Jack: are you interested in making botan compliant to standard configure/make scenario ? I know it's overkill sometimes but making configuration script names configure and supporting --prefix option would be very convienient for some ad-hoc builders. That's exactly the line of thinking I'd like to encourage: if our dependent libraries don't build on other systems, help them with their build system. That's much more in the open source spirit than coming up with our own one - as we currently do. (Although I think botan is a bad example here, because its build harness looks quite thought through and tested on several platforms - certainly more than monotone currently has). PS. I would like to test this library-build branch on mingw, is it buildable at least on some common platform like linux ? Checkout the branch net.venge.monotone.library-build. Expect having to fiddle with autoconf et al. Corrections and improvements very welcome. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
2008/9/29 Jack Lloyd [EMAIL PROTECTED]: On Mon, Sep 29, 2008 at 12:01:50PM +0200, Zbigniew Zag??rski wrote: Jack: are you interested in making botan compliant to standard configure/make scenario ? Um not particularly, TBH. I really really do not like autoconf. And it is completely useless off Unix. Have very similar feelings. However most liked feature for me is ability automagically detect cross-compiling environment. That said if someone sent me a configure.in that worked and handled the same things configure.pl does I would certainly ship it at least as an option. I know it's overkill sometimes but making configuration script names configure and supporting --prefix option would be very convienient for some ad-hoc builders. Sorry I was misguided by fact that Makefile contained hardocoded C:\Botan ... i didn't notice that it was generated. Just it being named configure instead of configure.pl is a major factor? :( No, i was struggling with --os --cpu for a moment (wanting to build asm versions) but now everything is fine. BTW, I've got patch that enables configure.pl to detect mingw/msys automagically. -- Zbigniew Zagórski / software developer / geek / happy daddy / ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
On Tue, Sep 30, 2008 at 02:13:47PM +0200, Zbigniew Zag??rski wrote: 2008/9/29 Jack Lloyd [EMAIL PROTECTED]: On Mon, Sep 29, 2008 at 12:01:50PM +0200, Zbigniew Zag??rski wrote: Jack: are you interested in making botan compliant to standard configure/make scenario ? Um not particularly, TBH. I really really do not like autoconf. And it is completely useless off Unix. Have very similar feelings. However most liked feature for me is ability automagically detect cross-compiling environment. That is an advantage, must admit. Really, I just don't underststand m4 and most my dislike of autoconf/automake stems from that probably. Well, some at least. ;) but now everything is fine. BTW, I've got patch that enables configure.pl to detect mingw/msys automagically. Saw it, thanks! -Jack ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
Is the idea of the library build branch to have monotone to build all of the libraries it needs, but do this by using each library's own build system or is it to get monotone to a state where it requires and uses system installed libraries? I was playing around with the idea of using my system installed botan the other day and it wasn't very hard to get monotone to use what I have already installed, although I haven't added any autoconf checks to see what version of botan is available and whether it is suitable or not. I would expect these would be simple enough to add for someone who has a clue about autoconf. Cheers, Derek ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
On Mon, Sep 29, 2008 at 9:27 AM, Jack Lloyd [EMAIL PROTECTED] wrote:
I know it's overkill sometimes but making configuration script
names configure and supporting --prefix option would be very
convienient for some ad-hoc builders.
Just it being named configure instead of configure.pl is a major
factor? :(
Supporting the same command line interface as GNU configure is a major
factor, because that means higher level scripts don't have to know
that botan is different. I'd recommend you do two things:
1) Create a shell script named configure and a DOS batch file named
configure.bat that wrap the existing configure.pl; they can be as
simple as
#! /bin/sh
exec perl -- ${0%/*}/configure.pl $@
(I don't know the equivalent in dos batch language.)
2) Make the command line options supported by configure.pl match the
command line options supported by autoconf-generated configure scripts
as closely as possible, and make the generated Makefile use the same
targets and user-settable variables as automake-generated Makefiles.
Looking at configure.pl --help I would suggest:
--no-foo - --disable-foo, for all foo; also, accept --enable-foo
for all foo, even if this has no effect
--debug - --enable-debug (and accept --disable-debug)
--modules, --module-set=, --noauto - collapse into
--(enable,disable)-modules
--build-dir= - --with-build-dir; also, if it doesn't already, this
should default to the current working directory
*even if* that's not the same as the directory
containing configure.pl; also, support --srcdir=
DESTDIR= support for make install, if not already present
--cc= - CC environment variable and/or CC=compiler on the command line
--os, --cpu - --host=GNU configuration triplet
--endian=, --unaligned-mem= should be spelled --with-endian,
--with-unaligned-mem
--local-config= - --with-local-config=
also, it would be nice (but is not very important) to support
--exec-prefix and as many of the --foodir installation options as
possible (accept them even if you don't have anything to put in that
directory).
in roughly descending order of importance, with everything up to the
gap much more important than everything below. Note that you can
include config.sub and config.guess in your project even if it doesn't
use autoconf, and with no impact on licensing of anything else. I
don't know offhand what changes might be needed for the makefiles.
zw
___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
On Tue, Sep 30, 2008 at 8:55 AM, Derek Scherger [EMAIL PROTECTED] wrote: Is the idea of the library build branch to have monotone to build all of the libraries it needs, but do this by using each library's own build system or is it to get monotone to a state where it requires and uses system installed libraries? The original idea was to have it build all of the libraries it needs using each library's own build system, with the option to use system installed libraries if desired by the builder. I got hideously stuck on this because it really needs to be set up in a way that automake and autoconf aren't designed for -- someone mentioned upthread that it's bizarre to invoke subdirectory configure scripts from the makefile, and yes it is, but if this is what we want, *there is no alternative*, because AC_CONFIG_SUBDIRS does not allow you to modify the command line options. It also became clear that there are still quite a few hacks that we rely on in our local copies of the libraries. The big one was, we compile lua as C++ so its error handling turns into C++ exceptions. A system-provided liblua won't do this. We don't do error propagation across the lua/C++ interface correctly anyway (errors in lua hooks have a bad way of just disappearing), so I would *like* to rip that out and go back to lua's C interface, but I haven't had time. It might make sense to start a new project to rip out our bundled libraries altogether and then fix everything necessary to make it work with system libraries. In fact, I think that might be a good stepping stone to a more sane way of bundling libraries, even if we do want to keep doing that. If we do start that project, though, it should use a different branch than library-build, because they're very different schemes and they'll trip over each other no end. zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
2008/9/28 Zbigniew Zagórski [EMAIL PROTECTED]: 2008/9/28 Stephen Leake [EMAIL PROTECTED]: ... MinGW doesn't have a decent package manager, and I don't know where to get pcre for MinGW; I looked for that a while ago. I haven't looked for the others. AFAIR, on properly configured mingw+msys ./configure --prefix=/mingw make install works like a charm for pcre-7.7. However, i don't know if botan sqlite builds natively on mingw32 without tricks. I'll try to pefrorm some research on topic. From my recent tests it appears that: - lua - pcre - botan - sqlite3 build without any problem on mingw/msys with simple ./configure --prefix=/mingw ; make ; make install However, lua and botan need some custom configuration and makefile invocation. I failed to build libidn on mingw and gave up after some strange libtool+DLL linking errors. Jack: are you interested in making botan compliant to standard configure/make scenario ? I know it's overkill sometimes but making configuration script names configure and supporting --prefix option would be very convienient for some ad-hoc builders. PS. I would like to test this library-build branch on mingw, is it buildable at least on some common platform like linux ? -- Zbigniew Zagórski / software developer / geek / happy daddy / ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
Zbigniew Zagórski [EMAIL PROTECTED] writes: 2008/9/28 Stephen Leake [EMAIL PROTECTED]: Zbigniew Zagórski [EMAIL PROTECTED] writes: 2008/9/28 Stephen Leake [EMAIL PROTECTED]: ... I agree this is a reasonable goal, at least on platforms that have decent package managers. MinGW doesn't have a decent package manager, and I don't know where to get pcre for MinGW; I looked for that a while ago. I haven't looked for the others. AFAIR, on properly configured mingw+msys properly is of course the issue. Well, the only changes i have is custom m4, autoconf, automake a result of private battle in order to get recent versions of these. Rest is standard ... i think. This monotone wiki page: http://www.venge.net/mtn-wiki/BuildingOnWindows has instructions for installing the mingw tools needed to build mtn. Can you test with a setup following those instructions, and add the pcre etc instructions? Where did you get the mingw pcre package from? I've used source found on http://pcre.org/, normal download no port needed - 7.7, 7.8 versions build. Ok, that's good. -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
On Mon, Sep 29, 2008 at 12:01:50PM +0200, Zbigniew Zag??rski wrote: Jack: are you interested in making botan compliant to standard configure/make scenario ? Um not particularly, TBH. I really really do not like autoconf. And it is completely useless off Unix. That said if someone sent me a configure.in that worked and handled the same things configure.pl does I would certainly ship it at least as an option. I know it's overkill sometimes but making configuration script names configure and supporting --prefix option would be very convienient for some ad-hoc builders. Just it being named configure instead of configure.pl is a major factor? :( The --prefix option is supported BTW, though install on Windows IIRC does not work. -Jack ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
On Sun, Sep 28, 2008 at 02:52:14PM +0200, Zbigniew Zag??rski wrote: However, i don't know if botan sqlite builds natively on mingw32 without tricks. I'll try to pefrorm some research on topic. Recent 1.7 releases has been tested on MinGW with GCC 3.4.5, no problems. Let me know if you encounter any issues. -Jack ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
Zbigniew Zagórski [EMAIL PROTECTED] writes: 2008/9/28 Stephen Leake [EMAIL PROTECTED]: Markus Wanner [EMAIL PROTECTED] writes: Instead we should IMO simply rip out the still-bundled stuff (pcre, sqlite, idna, botan) and adjust the Makefile cruft to build against system provided libraries - as the vast majority of other software does. Or how often did you encounter a 'configure' which doesn't configure anything but leaves that task for 'make', making you fiddle with aclocal in subdirectories if things don't work out as planned? Comments? I agree this is a reasonable goal, at least on platforms that have decent package managers. MinGW doesn't have a decent package manager, and I don't know where to get pcre for MinGW; I looked for that a while ago. I haven't looked for the others. AFAIR, on properly configured mingw+msys properly is of course the issue. Where did you get the mingw pcre package from? -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] library build
Hi, I've taken out the library-build branch again today and played around a bit. It took me several approaches and fiddling with auto* just to get it to compile again. I still feel uncomfortable maintaining a special wrapper and multiple 3rd party libraries inside monotone with the additional option to use a system provided library. It seems obvious to me that we don't have the manpower to maintain such a multi-purpose build harness, so I'm proposing to go the easier way and make monotone dependent on 3rd party libraries, without providing them (and a kitchen-sink build harness). Instead we should IMO simply rip out the still-bundled stuff (pcre, sqlite, idna, botan) and adjust the Makefile cruft to build against system provided libraries - as the vast majority of other software does. Or how often did you encounter a 'configure' which doesn't configure anything but leaves that task for 'make', making you fiddle with aclocal in subdirectories if things don't work out as planned? Comments? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
Markus Wanner [EMAIL PROTECTED] writes: Instead we should IMO simply rip out the still-bundled stuff (pcre, sqlite, idna, botan) and adjust the Makefile cruft to build against system provided libraries - as the vast majority of other software does. Or how often did you encounter a 'configure' which doesn't configure anything but leaves that task for 'make', making you fiddle with aclocal in subdirectories if things don't work out as planned? Comments? I agree this is a reasonable goal, at least on platforms that have decent package managers. MinGW doesn't have a decent package manager, and I don't know where to get pcre for MinGW; I looked for that a while ago. I haven't looked for the others. Maybe the upstream sources are adequate for MinGW; I have not looked. I guess that's where we get our bundled sources from, anyway. We need to continue to support MinGW. If we can find sources of all the required packages, we can just list those in INSTALL (and on the wiki page). Maybe configure can suggest reading INSTALL if it doesn't find the packages. If we can't find sources for the packages, maybe we can have a simpler MinGW-only build for bundled packages? -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build progress report and request for help with botan
Hi, I've finally got around playing with that branch again. Some remarks below. Zack Weinberg wrote: Monotone itself has been moved to another subdirectory (currently named monotone, but I could be persuaded to rename it src or something like that). +1 for naming it src. The top-level configure script just saves all its arguments into a Makefile. That Makefile invokes the library configure scripts with --prefix set to a subdirectory libinst of the build tree, then builds them and installs them. It then runs the monotone-subdirectory configure script, telling it to look in libinst for its libraries. Trying that setup, I've been puzzled about all the steps needed. I still don't get warm with the mixture of the configuration step and the actual building step. However, that might just be personal preference. Trying to build the whole thing, I've encountered the following problems: * I've spent some time to figure out that I needed to run aclocal for every subdir. Only later on, I've noticed that there's an autogen.sh script which does that for me. However, as INSTALL still recommends running aclocal manually, that might entrap other users as well. * If pcre's configure script fails after having created pcre/Makefile (for example due to missing config.h.in), a subsequent 'make pcre/Makefile' won't restart configure for pcre, because the Makefile is up-to-date. Maybe simply add more of the files there as dependencies, so that make reruns pcre's configure. * Botan's 'make install' complains about ownership problems: install: cannot change ownership of `/home/markus/projects/monotone/build.nvm.library-build/libinst/share/doc/Botan-1.7.4/credits.txt': Operation not permitted Do we really need to run 'make install' for these 3rdparty libraries? Having top-level 'make' recurse into all of subdirs running 'make install' every time seems expensive to me. After having worked around these issues, I've been able to run 'make' and 'make install' in all of the subdirectories (lua, botan, pcre, netxx, idna and sqlite). Don't even try building the monotone subdirectory yet. People wanting to experiment may find the toplevel targets make dir/Makefile and make libinst/dir-stamp of use. Jup, even the -stamp targets are working. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build progress report and request for help with botan
On Wed, Apr 9, 2008 at 1:02 PM, Markus Schiltknecht [EMAIL PROTECTED] wrote: Trying that setup, I've been puzzled about all the steps needed. I still don't get warm with the mixture of the configuration step and the actual building step. However, that might just be personal preference. It is unusual, but I see no alternative, given that we must be able to control the command line switches passed to subdirectory configure scripts, and AC_CONFIG_SUBDIRS has no support for this. (Also, AC_CONFIG_SUBDIRS does not know how to run botan's configure.pl correctly.) I also think that the configure script for the monotone subdirectory will be simpler if it is run after the (enabled subset of the) bundled libraries are installed to the staging directory, but as I haven't yet revised that script, I don't know for sure. * I've spent some time to figure out that I needed to run aclocal for every subdir. Only later on, I've noticed that there's an autogen.sh script which does that for me. However, as INSTALL still recommends running aclocal manually, that might entrap other users as well. I intended to edit INSTALL only after everything was done, but we can certainly change it now if you think that's more useful. (You mean autoreconf, right?) * If pcre's configure script fails after having created pcre/Makefile (for example due to missing config.h.in), a subsequent 'make pcre/Makefile' won't restart configure for pcre, because the Makefile is up-to-date. Maybe simply add more of the files there as dependencies, so that make reruns pcre's configure. I didn't do that only because I get a headache if I try to write down the set of rules that would be necessary, and I thought I could get away with not doing it yet. * Botan's 'make install' complains about ownership problems: Jack's fixed this upstream, but for now you can do 'make install USER=u GROUP=g' where u and g are your username and primary group, respectively. I mean to put that into the top-level Makefile until a new botan comes out. Do we really need to run 'make install' for these 3rdparty libraries? Yes. If we instead use -I to pick up headers from the source tree, builds with the bundled libraries will be able to see library-internal headers, whereas builds with system libraries won't; this is asking for trouble. Also, running 'make install' means that the monotone subdirectory doesn't have to know anything about the structure of the bundled library subdirectories; it just does -I libinst/include -L libinst/lib and it's happy. Having top-level 'make' recurse into all of subdirs running 'make install' every time seems expensive to me. This is what libinst/*-stamp are for. A second 'make' at top level shouldn't recurse. Of course, we need real dependencies for those so that it does rebuild a subdirectory that's changed, but again, that is something I thought I could get away with implementing later. After having worked around these issues, I've been able to run 'make' and 'make install' in all of the subdirectories (lua, botan, pcre, netxx, idna and sqlite). Cool beans. Did you get the autoconf tests that are for sqlite's benefit out of monotone/configure.ac and into the stub sqlite/configure.ac, or did you drop in the upstream sqlite build harness, or? A nice next-action would be to get botan and sqlite's internal testsuites imported and working. zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build progress report and request for help with botan
On Fri, Apr 04, 2008, Zack Weinberg wrote: nvm.library-build is my project to build all our bundled libraries in a way that more closely resembles how they're built upstream. (This will also facilitate optional use of system-provided versions.) It's intended to go like this: Each bundled library has its own subdirectory, just as they do now, but those subdirs have Makefiles and configure scripts of their own. Monotone itself has been moved to another subdirectory (currently named monotone, but I could be persuaded to rename it src or something like that). The top-level configure script just saves all its arguments into a Makefile. That Makefile invokes the library configure scripts with --prefix set to a subdirectory libinst of the build tree, then builds them and installs them. It then runs the monotone-subdirectory configure script, telling it to look in libinst for its libraries. I'm not convinced whether the install step here is really necessary here. About a year ago, for RPM 5 I've implemented a rather sophisticated reusable Autoconf macro RPM_CHECK_LIB (see attachment) which allows us to build against the dozen third-party libraries RPM requires in both internal/embedded and external variant. We put really lots of efforts into this scheme in 2007 and it showed up that it works very good in practice. And we do not require the library to be really installed here (as this usually increases the disk space requirements during build-time even more). Perhaps this approach could be also useful for Monotone here. Right now what works is building idna, lua, netxx, and pcre. My original intent was to have these be verbatim upstream tarballs but that proved impractical, I've pretty much written my own Makefiles and configure.ac's for all of them. [Only idna and pcre have upstream build harnesses that are any use, and they both have tons of unrelated junk that we don't want cluttering up the repository.] I think if you don't have to install the libraries you usually can use the Makefile's out-of-the-box and really keep the stuff fully verbatim. botan builds but does not install, because we are using the upstream build harness for that, and it wants files that were left out of the original import. I would appreciate help from people who've done upstream botan imports on that front. sqlite doesn't build - I just haven't gotten to that one yet. I'd like to know, though, whether we want to move to sqlite 3.5.7 separately or as part of this project (looking at the big picture, my inclination is to say separately, but doing it as part of this project might make this project a wee easier - you'll note that I updated the idna directory, where the tradeoff is clearer). Current status of my evaluation SQLite 3.5.x for Monotone is: | Current branch: net.venge.monotone | Changes against parent 36b6cb016019e675bc24a2fa97d4eb19fa4919af | dropped sqlite | dropped sqlite/alter.c | dropped sqlite/analyze.c | dropped sqlite/attach.c | dropped sqlite/auth.c | dropped sqlite/btree.c | dropped sqlite/btree.h | dropped sqlite/btreeInt.h | dropped sqlite/build.c | dropped sqlite/callback.c | dropped sqlite/complete.c | dropped sqlite/date.c | dropped sqlite/delete.c | dropped sqlite/expr.c | dropped sqlite/func.c | dropped sqlite/hash.c | dropped sqlite/hash.h | dropped sqlite/insert.c | dropped sqlite/keywordhash.h | dropped sqlite/legacy.c | dropped sqlite/loadext.c | dropped sqlite/main.c | dropped sqlite/malloc.c | dropped sqlite/opcodes.c | dropped sqlite/opcodes.h | dropped sqlite/os.c | dropped sqlite/os.h | dropped sqlite/os_common.h | dropped sqlite/os_os2.c | dropped sqlite/os_os2.h | dropped sqlite/os_unix.c | dropped sqlite/os_win.c | dropped sqlite/pager.c | dropped sqlite/pager.h | dropped sqlite/parse.c | dropped sqlite/parse.h | dropped sqlite/pragma.c | dropped sqlite/prepare.c | dropped sqlite/printf.c | dropped sqlite/random.c | dropped sqlite/select.c | dropped sqlite/sqlite3.h | dropped sqlite/sqlite3ext.h | dropped sqlite/sqliteInt.h | dropped sqlite/sqliteLimit.h | dropped sqlite/table.c | dropped sqlite/tokenize.c | dropped sqlite/trigger.c | dropped sqlite/update.c | dropped sqlite/utf.c | dropped sqlite/util.c | dropped sqlite/vacuum.c | dropped sqlite/vdbe.c | dropped sqlite/vdbe.h | dropped sqlite/vdbeInt.h | dropped sqlite/vdbeapi.c | dropped sqlite/vdbeaux.c | dropped sqlite/vdbeblob.c | dropped sqlite/vdbefifo.c | dropped sqlite/vdbemem.c | dropped sqlite/vtab.c | dropped sqlite/where.c | addedsqlite3.c | addedsqlite3.h | patched Makefile.am | patched database.cc | patched m4/sqlite.m4 | patched schema_migration.cc | patched tests/fail_cleanly_on_unreadable_db/__driver__.lua As you see, I've replaced the sqlite/* subdir with the new amalgamation
Re: [Monotone-devel] library-build progress report and request for help with botan
Hi, Zack Weinberg wrote: On Sat, Apr 5, 2008 at 12:37 PM, Zack Weinberg [EMAIL PROTECTED] wrote: On Sat, Apr 5, 2008 at 6:41 AM, Markus Schiltknecht [EMAIL PROTECTED] wrote: I was trying to keep the current, stripped botan variant, so as to keep changes to a minimum. That one gets propagated from ... uhm... au...matthew.something.somewhere... see botan/README.monotone, I've updated comments in there. I'll look at that. I *think* the fix is simple - it appears to just want one more file - but I was a little scared of the merge process last night. First of all, have you pulled from my server, nabagan.bluegap.ch? AFAICT that's the only place hosting the newest revisions of that branch. (As I couldn't push to the original server, nor monotone.ca). If we stick to that special and troublesome stripped botan variant, we should first make sure we all have access to that branch. Richard, can you organize write permissions to au.asn.ucc.matt.botan.monotone-2 for at least me and Zack on monotone.ac? Or shall we better rename that branch to nvm.something...? OTOH, that would pull the whole botan repository into the monotone namespace (i.e. net.venge.monotone*) which is what Matthew was trying to avoid. Ok, I've confirmed that the major problem with the botan subdirectory is that readme.txt is missing, but I'm not 100% sure I understand how to pull in things from au.asn.ucc.matt.botan.monotone-2 ... the file is in the Attic there. Before I do anything I would appreciate it if someone could confirm that this is the right approach. The Attic is to avoid die die die merge problems. Just move readme.txt from the Attic back to it's original location. That way, propagating from net.randombit.botan keeps working as expected. 1) in aaumb, mtn mv Attic/readme.txt readme.txt ; commit. 2) copy the file from the aaumb checkout to a nvm.botan checkout's botan subdirectory, mtn add it, and add it to EXTRA_DIST in the top-level makefile. Commit. 3) Propagate from nvm.botan to nvm. 4) Propagate from nvm to nvm.library-build. (I think the file will just magically appear, because the botan subdirectory in nvm.library-build has the same inode as the one in nvm... am I right?) That looks correct. You could even skip propagating to nvm and instead propagate from nvm.botan to nvm.library-build directly. This updating process is rather expensive in terms of work needed for an upgrade of the provided botan version. I was hoping that we can replace that process with simply dropping a tarball of the newest botan library into our botan/ subdirectory and automate the rest. Of course, if the build process (configure.pl and make) would change, we'd still have to adapt. But that's much less work than having to manually add, replace or delete files around, as I had to do in the past. So, opposed to the other subprojects, I'd now vote for removing our stripped botan variant, including that staging branch (the aaumb one) and land a (full) botan 1.7.4 there, for the sake of simplifying later upgrades. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build progress report and request for help with botan
On Sun, Apr 6, 2008 at 9:40 AM, Markus Schiltknecht [EMAIL PROTECTED] wrote: First of all, have you pulled from my server, nabagan.bluegap.ch? AFAICT that's the only place hosting the newest revisions of that branch. (As I couldn't push to the original server, nor monotone.ca). Yes, I have. If we stick to that special and troublesome stripped botan variant For the record, I only want to do that for the initial work on this branch. After it's done and merged I think an update to a full copy of botan should be very high on the priority list, and I'm good with just using tarballs. Thanks for the explanation of how to do the one-file addition. zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build progress report and request for help with botan
Hi, Zack Weinberg wrote: Right now what works is building idna, lua, netxx, and pcre. My original intent was to have these be verbatim upstream tarballs but that proved impractical, I've pretty much written my own Makefiles and configure.ac's for all of them. [Only idna and pcre have upstream build harnesses that are any use, and they both have tons of unrelated junk that we don't want cluttering up the repository.] Well, if cluttering up the repository is the only counter argument, I'd say better use their build harnesses, because maintenance and upgrading becomes much simpler. Otherwise, we will have to maintain our own build harness for that, which is about as bad as it is today. But it sounds like you hit other problems as well. I'm just pointing out the maintenance aspect again... botan builds but does not install, because we are using the upstream build harness for that, and it wants files that were left out of the original import. I would appreciate help from people who've done upstream botan imports on that front. I was trying to keep the current, stripped botan variant, so as to keep changes to a minimum. That one gets propagated from ... uhm... au...matthew.something.somewhere... see botan/README.monotone, I've updated comments in there. But maybe it's not worth maintaining that upgrading strategy. Instead, let's just replace it with the original tarball. It's the same 'cluttering' vs 'simple maintenance' decision: we currently have a sleek and stripped botan for monotone. I didn't dare giving that up for simpler maintenance, yet. But in the long run, we *should* replace it with the tarball, I think. Cluttering the repository is much less expensive for us, than having to clean up our own test harness for every botan release. sqlite doesn't build - I just haven't gotten to that one yet. I'd like to know, though, whether we want to move to sqlite 3.5.7 separately or as part of this project (looking at the big picture, my inclination is to say separately, but doing it as part of this project might make this project a wee easier - you'll note that I updated the idna directory, where the tradeoff is clearer). I'd certainly vote for decoupling these changes, because it might actually be *less* work in the end, due to simpler testing and debugging. Don't even try building the monotone subdirectory yet. People wanting to experiment may find the toplevel targets make dir/Makefile and make libinst/dir-stamp of use. This sounds like you've displaced autoconf somewhat. I've thought it would be clearer if ./configure would also configure the sub directories. What were the reasons for doing that from make? Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build progress report and request for help with botan
On Sat, Apr 5, 2008 at 12:37 PM, Zack Weinberg [EMAIL PROTECTED] wrote: On Sat, Apr 5, 2008 at 6:41 AM, Markus Schiltknecht [EMAIL PROTECTED] wrote: I was trying to keep the current, stripped botan variant, so as to keep changes to a minimum. That one gets propagated from ... uhm... au...matthew.something.somewhere... see botan/README.monotone, I've updated comments in there. I'll look at that. I *think* the fix is simple - it appears to just want one more file - but I was a little scared of the merge process last night. Ok, I've confirmed that the major problem with the botan subdirectory is that readme.txt is missing, but I'm not 100% sure I understand how to pull in things from au.asn.ucc.matt.botan.monotone-2 ... the file is in the Attic there. Before I do anything I would appreciate it if someone could confirm that this is the right approach. 1) in aaumb, mtn mv Attic/readme.txt readme.txt ; commit. 2) copy the file from the aaumb checkout to a nvm.botan checkout's botan subdirectory, mtn add it, and add it to EXTRA_DIST in the top-level makefile. Commit. 3) Propagate from nvm.botan to nvm. 4) Propagate from nvm to nvm.library-build. (I think the file will just magically appear, because the botan subdirectory in nvm.library-build has the same inode as the one in nvm... am I right?) zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] library-build progress report and request for help with botan
nvm.library-build is my project to build all our bundled libraries in a way that more closely resembles how they're built upstream. (This will also facilitate optional use of system-provided versions.) It's intended to go like this: Each bundled library has its own subdirectory, just as they do now, but those subdirs have Makefiles and configure scripts of their own. Monotone itself has been moved to another subdirectory (currently named monotone, but I could be persuaded to rename it src or something like that). The top-level configure script just saves all its arguments into a Makefile. That Makefile invokes the library configure scripts with --prefix set to a subdirectory libinst of the build tree, then builds them and installs them. It then runs the monotone-subdirectory configure script, telling it to look in libinst for its libraries. Right now what works is building idna, lua, netxx, and pcre. My original intent was to have these be verbatim upstream tarballs but that proved impractical, I've pretty much written my own Makefiles and configure.ac's for all of them. [Only idna and pcre have upstream build harnesses that are any use, and they both have tons of unrelated junk that we don't want cluttering up the repository.] botan builds but does not install, because we are using the upstream build harness for that, and it wants files that were left out of the original import. I would appreciate help from people who've done upstream botan imports on that front. sqlite doesn't build - I just haven't gotten to that one yet. I'd like to know, though, whether we want to move to sqlite 3.5.7 separately or as part of this project (looking at the big picture, my inclination is to say separately, but doing it as part of this project might make this project a wee easier - you'll note that I updated the idna directory, where the tradeoff is clearer). Don't even try building the monotone subdirectory yet. People wanting to experiment may find the toplevel targets make dir/Makefile and make libinst/dir-stamp of use. zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] library-build branch
Hi, to get into autoconf et al, I've played around with the library-build branch. Hope Zack doesn't mind... :-) I've reverted botan to its own directory layout, using its configure.pl script. I'm not sure invocation of that perl script is correct in configure.ac, however, it seems to work. For everything else (idna, pcre, sqlite, lua), I've just added a handwritten Makefile.am. From here, that now compiles .a files for every library we use. To be able to really compile and test monotone itself, we still need to clean up between Makefile.am and monotone/Makefile.am. Also, adding more --with-system-$(library) switches would be useful, however, first of all, I'd like to make monotone compile again that way. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library-build branch
On Wed, Mar 26, 2008 at 7:02 PM, Markus Schiltknecht [EMAIL PROTECTED] wrote: Hi, to get into autoconf et al, I've played around with the library-build branch. Hope Zack doesn't mind... :-) cool. I had gotten seriously stuck on that, so I'm glad to see you picking up on it. I've not got much time for monotone in the next few days but when I do I'll lookat it. zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
