Status quo on my build environment anti-convolution efforts
Status Quo: The RPM build environment is really one of the most convoluted ones I've ever seen. And I've really seen _MANY_ nasty build environments during the last decade ;-) The RPM build environment is full of unclean variable overrides, dozends of hacks and more. Most of the stuff seems to be just left-overs from RPMs evolution during the last 10 years, others were just ugly hacks for Red Hat Linux platforms, others were done by accident, etc. Following my personal agenda (for more details just see my BLOG under http://trainofthoughts.org/blog/2007/06/05/rpm-relaunched/) my ultimate goal is to get RPM 5 building in a _clean_, _straight-forward_ and _consistent_ way under _all_ major Unix platforms, so it especially finally can be integrated also into the next version of the cross-platform software distribution OpenPKG (which currently is still using a hacked RPM 4.2.1). Although I've already completed most my step 5, the RPM build environment is such convoluted and spiked with special cases that I got thrown back to step 4 (modernize the Autoconf/Automake based build environment). Hence I'm still investigating how to further streamline the RPM build environment. After many dozends of both surgical and sledgehammer-style commits we already reached a state where the monster named configure.ac is in at least in an acceptable state again (all ugly hacks are at least bandaged) and the Makefile stuff is slowly approaching a cleaner state, too. Nevertheless, two major points still cause me headaches and hence I've to still investigate there further. A few details follow in case you are monitoring me ;-) 1. The configure.ac and make install code behaves slightly different on different platforms. This can be seen an advantage or a disadvantage. For me it is a disadvantage because I expect exactly the same results under all platforms. Hence, from my POV we at least need the possibility to explicitly _fixate_ the installation layout, etc. By default RPM could still use its current do it differently for various platforms approach, but IMHO we at least need the possibility to fine-tune at least all paths explicitly when needed. I'll investigate here and perhaps we need a bunch of additional Autoconf options. Reminds me somewhat about my efforts to the Apache 1.3 build environment where at the end I had to introduce a config.layout to make all parties happy. We'll see... 2. Although we finally have a sophisticated way to link against the third-party libraries, we still have a related problem: the internal copies of third-party libraries are linked differently against RPM. db is packed directly into librpmdb.la, lua by accident is packed into librpm.la, etc. All not consistent and hence confusing and on some platforms even causing problems (Jeff for instance got failures with the Lua linking). Here I've to investigate, too. I'm still not sure what the best approach is, but it should be clear that although the local copies of third-party libraries are not explicitly _installed_ (as libdb.la, liblua.la, etc) their _content_ has to be installed in some way or our librpm*.la are useless in practice. The approach of librpmdb.la to include libdb.la seems to be a ticky and acceptable solution although it has to be done for _all_ libraries and perhaps the third-party stuff should be grouped together (perhaps even in the new librpmmisc.la which is some sort of a container for remaining and dynamically selected stuff anyway). I'll look what I can do for us... Watch out for more commits... Ralf S. Engelschall [EMAIL PROTECTED] www.engelschall.com __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: rpm -qa speedup
On Jun 24, 2007, at 4:19 PM, Pawel A. Gajda wrote:
rpm -qa slowness (even with --no{digest,signature}) makes poldek
(suppose other tools too) uncomfortable in use. Although poldek
manages its own rpmdb cache, every installation/removal made by rpm
(and other tools) forces package database scanning for added/removed
items.
Um, add --stats and report byour measurements. Undocumented claims
of slowness help noone.
Simple solution could be additional package index with the only basic
(NEVR, ARCH, OS, digest) package meta information. It would
drastically speed up database scanning and make users happier.
Users will never be happy.
Jeff, does such PackagesLite make sense for you? It would be an issue
to add it? Would you accept the patch? If yes, maybe have you an idea
how to fit such lite index into rpm rpmdb*Iterator() architecture? Or
maybe just create rpmdbLitePackagesIterator() which returns stripped
down Headers?
See /var/cache/hrmib. The results of rpm -qa in 0 length files ready to
be read using opendir(3).
73 de Jeff
__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org
Re: isCompressed reports tarballs to be LZMA files
On Jun 24, 2007, at 5:50 PM, Alex Myltsev wrote: In ALT Linux we often use uncompressed tarballs as RPM Source files. RPM's LZMA compression support breaks our packages: in many cases, the tarball will contain zeroes at offsets 9..12, and isCompressed() will report it to be an LZMA-compressed file. One possible fix is to use libmagic, which seems to be able to tell tarballs from LZMA compression. Alternatively, we could check for an .lzma extension in the filename. (I could just disable LZMA decompression support in our build, but I don't want to create local incompatibilities if 4.5 is going to be the all-round compatible version of RPM.) I suspect that libmagic just detects tarballs first, which is not what isCompressed() is concerned with. No .lzma suffix detection please. Disabling until LZMA gets magic is likely the best solution imho. There are several flaws in the lzma - rpm implementation, not the least of which is the copyright spew every time the helper is invoked. 73 de Jeff __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: LP64 alignment issue in 4.4.9 and CVS
On Jun 26, 2007, at 12:28 PM, Tim Mooney wrote: rpm 4.4.9 on sparcv9-sun-solaris2.8 (LP64) compiled with the Sun Workshop 11 compiler gets a reproduceable SIGBUS when installing a package or importing a key with a fresh database. I haven't seen this problem on my x86_64-sun-solaris2.10 workstation which is also running rpm 4.4.9. x86_64 must be more forgiving of alignment than LP64 sparc. This bug was reported by PLD/sparc64 abt a week ago. There's an alignment issues with hcolor. See recent checkin for fix on both HEAD and rpm-4_5. hth 73 de Jeff __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: LP64 alignment issue in 4.4.9 and CVS
In regard to: Re: LP64 alignment issue in 4.4.9 and CVS, Jeff Johnson said...: On Jun 26, 2007, at 12:28 PM, Tim Mooney wrote: rpm 4.4.9 on sparcv9-sun-solaris2.8 (LP64) compiled with the Sun Workshop 11 compiler gets a reproduceable SIGBUS when installing a package or importing a key with a fresh database. I haven't seen this problem on my x86_64-sun-solaris2.10 workstation which is also running rpm 4.4.9. x86_64 must be more forgiving of alignment than LP64 sparc. This bug was reported by PLD/sparc64 abt a week ago. There's an alignment issues with hcolor. See recent checkin for fix on both HEAD and rpm-4_5. Thanks for the info, sorry for the duplicate report. Tim -- Tim Mooney[EMAIL PROTECTED] Information Technology Services (701) 231-1076 (Voice) Room 242-J6, IACC Building(701) 231-8541 (Fax) North Dakota State University, Fargo, ND 58105-5164 __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: LP64 alignment issue in 4.4.9 and CVS
On Jun 27, 2007, at 2:55 PM, Tim Mooney wrote: This bug was reported by PLD/sparc64 abt a week ago. There's an alignment issues with hcolor. See recent checkin for fix on both HEAD and rpm-4_5. Thanks for the info, sorry for the duplicate report. Heh, thanks for reporting even if duplicate. The bug has been there for a while, easy fixing when reported. The problem is seen only on sparc64. 73 de Jeff __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: [CVS] RPM: rpm/ autogen.sh rpm/rpmdb/ Makefile.am
On Jun 27, 2007, at 3:20 PM, Ralf S. Engelschall wrote: RPM Package Manager, CVS Repository http://rpm5.org/cvs/ __ __ Server: rpm5.org Name: Ralf S. Engelschall Root: /v/rpm/cvs Email: [EMAIL PROTECTED] Module: rpm Date: 27-Jun-2007 21:20:45 Branch: HEAD Handle: 2007062720204500 Modified files: rpm autogen.sh rpm/rpmdb Makefile.am Log: In rpmdb/ the various Berkeley-DB utilities db_xxx are rebuild for installation purposes, although they got already built in db3/. As the utility object list is already incomplete (the building just still worked as librpm.la is linked in which contains all(!) Berkeley-DB objects anyway) and we don't need a second building of the tools, replace the building with simple symlinking. Also sort the utilities the same way in the _PROGRAMS list and in the targets list. This isn't the right fix. The system db (and utilities) may be different than the db (and utilities) used by rpm. It's the linkage against -lrpmdb that is important imho. 73 de Jeff __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: [CVS] RPM: rpm/ autogen.sh rpm/rpmdb/ Makefile.am
On Wed, Jun 27, 2007, Jeff Johnson wrote: [...] In rpmdb/ the various Berkeley-DB utilities db_xxx are rebuild for installation purposes, although they got already built in db3/. As the utility object list is already incomplete (the building just still worked as librpm.la is linked in which contains all(!) Berkeley-DB objects anyway) and we don't need a second building of the tools, replace the building with simple symlinking. Also sort the utilities the same way in the _PROGRAMS list and in the targets list. This isn't the right fix. The system db (and utilities) may be different than the db (and utilities) used by rpm. It's the linkage against -lrpmdb that is important imho. Yes, I know. But nothing has changed here. The installed db_xxx utilities are _still_ those built against RPM's internal Berkeley-DB. I just install the ones in db3/ and not the re-linked ones in rpmdb/. But as librpmdb contained just copies of the objects from db3/ the effective resulting db_xxx tools are exactly the same. Ralf S. Engelschall [EMAIL PROTECTED] www.engelschall.com __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Status quo on my build environment anti-convolution efforts
On Wed, Jun 27, 2007, Ralf S. Engelschall wrote: [...] 2. Although we finally have a sophisticated way to link against the third-party libraries, we still have a related problem: the internal copies of third-party libraries are linked differently against RPM. db is packed directly into librpmdb.la, lua by accident is packed into librpm.la, etc. All not consistent and hence confusing and on some platforms even causing problems (Jeff for instance got failures with the Lua linking). Here I've to investigate, too. I'm still not sure what the best approach is, but it should be clear that although the local copies of third-party libraries are not explicitly _installed_ (as libdb.la, liblua.la, etc) their _content_ has to be installed in some way or our librpm*.la are useless in practice. The approach of librpmdb.la to include libdb.la seems to be a ticky and acceptable solution although it has to be done for _all_ libraries and perhaps the third-party stuff should be grouped together (perhaps even in the new librpmmisc.la which is some sort of a container for remaining and dynamically selected stuff anyway). I'll look what I can do for us... After many hours thinking backward and forward, poking around in Automake documentation and performing stand-alone tests under FreeBSD 6 and Fedora 7 I finally think I've found a both clean, straight-forward and still fully portable solution to this mess. Berkeley-DB is now handled equally to all other internal third-party libs like Lua, File and ZLib and installed via a central bundling into librpmmisc. And the build-environment now no longer mixes the build flags of external and internal libraries (what caused the initial build problems for Jeff after RPM_CHECK_LIB was introduced). See http://rpm5.org/cvs/chngview?cn=7558 for details. I was now again able to cleanly build and install and test-run RPM 5 under FreeBSD 6 and Fedora 7 under both --enable-shared and --disable-shared and with and without a mixture of internal and external libraries. Under FreeBSD 6 I mostly built with --disable-shared and with dozend of uninstalled external libraries while under Fedora 7 I used system-installed versions for all the external libraries and built RPM with --enable-shared. Everything wors fine (again) now -- but this time we have a larger bunch of nasty hacks less in the build environment and a lot of new flexibility as my different tests under FreeBSD 6 and Fedora 7 show ;-) Ralf S. Engelschall [EMAIL PROTECTED] www.engelschall.com __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Status quo on my build environment anti-convolution efforts
This isn't what I would call glamorous work (I share JBJ's desire to know as little about autoconf as possible) but it doesn't have to be thankless. So I at least would like to publicly thank you for this tedious but important work. Here's to progress. :-) Jason On 6/27/07, Ralf S. Engelschall [EMAIL PROTECTED] wrote: On Wed, Jun 27, 2007, Ralf S. Engelschall wrote: [...] 2. Although we finally have a sophisticated way to link against the third-party libraries, we still have a related problem: the internal copies of third-party libraries are linked differently against RPM. db is packed directly into librpmdb.la, lua by accident is packed into librpm.la, etc. All not consistent and hence confusing and on some platforms even causing problems (Jeff for instance got failures with the Lua linking). Here I've to investigate, too. I'm still not sure what the best approach is, but it should be clear that although the local copies of third-party libraries are not explicitly _installed_ (as libdb.la, liblua.la, etc) their _content_ has to be installed in some way or our librpm*.la are useless in practice. The approach of librpmdb.la to include libdb.la seems to be a ticky and acceptable solution although it has to be done for _all_ libraries and perhaps the third-party stuff should be grouped together (perhaps even in the new librpmmisc.la which is some sort of a container for remaining and dynamically selected stuff anyway). I'll look what I can do for us... After many hours thinking backward and forward, poking around in Automake documentation and performing stand-alone tests under FreeBSD 6 and Fedora 7 I finally think I've found a both clean, straight-forward and still fully portable solution to this mess. Berkeley-DB is now handled equally to all other internal third-party libs like Lua, File and ZLib and installed via a central bundling into librpmmisc. And the build-environment now no longer mixes the build flags of external and internal libraries (what caused the initial build problems for Jeff after RPM_CHECK_LIB was introduced). See http://rpm5.org/cvs/chngview?cn=7558 for details. I was now again able to cleanly build and install and test-run RPM 5 under FreeBSD 6 and Fedora 7 under both --enable-shared and --disable-shared and with and without a mixture of internal and external libraries. Under FreeBSD 6 I mostly built with --disable-shared and with dozend of uninstalled external libraries while under Fedora 7 I used system-installed versions for all the external libraries and built RPM with --enable-shared. Everything wors fine (again) now -- but this time we have a larger bunch of nasty hacks less in the build environment and a lot of new flexibility as my different tests under FreeBSD 6 and Fedora 7 show ;-) Ralf S. Engelschall [EMAIL PROTECTED] www.engelschall.com __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Status quo on my build environment anti-convolution efforts
On Wed, Jun 27, 2007, Jason Corley wrote: This isn't what I would call glamorous work (I share JBJ's desire to know as little about autoconf as possible) but it doesn't have to be thankless. So I at least would like to publicly thank you for this tedious but important work. Here's to progress. :-) Well, thanks. It perhaps is a little bit similar to what Paul-Henning Kamp had to do when he introduced GEOM into FreeBSD in October 2002. He also had to cleanup first many really ugly things which people for years avoided to touch. In his The official 'GEOM is in the tree' speech he said: Road-construction is never a nice thing, at least not until it is done and over with, but it is a necessary part of the lifecycle. H... ;-) Ralf S. Engelschall [EMAIL PROTECTED] www.engelschall.com __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: [CVS] RPM: rpm/db3/ configure
This actually points out a bug I found the other day, but haven't been able to figure out how to solve. Using echo $* in this way causes variable or argument expansions that include a space to fail. We have a few cases where we need to pass in CFLAGS w/ multiple elements or CC='distcc ccache foo-gcc -te500v1'.. which causes a failure to occur. I replaced the args bit with a simple $@ in the args list to work around the issue.. but I don't know of a way of modifying the arguments in the shell and passing them properly quoted. --Mark patch -p0 '@@ .' Index: rpm/db3/configure $ cvs diff -u -r1.31 -r1.32 configure --- rpm/db3/configure 27 Jun 2007 18:05:12 - 1.31 +++ rpm/db3/configure 27 Jun 2007 21:18:48 - 1.32 @@ -9,6 +9,8 @@ ARGS=`echo $* |\ sed -e 's% [^ ]*CFLAGS=[^ ]*%%' \ + -e 's% [^ ]*CC=[^ ]*ccache [^ ]*%%' \ + -e 's% [^ ]*CC=[^ ]*%%' \ -e 's% -[^-][^ ]*%%g' \ -e 's% --param=[^ ]*%%g' \ -e 's%--cache-file=.*$%%' \ @@ . __ RPM Package Managerhttp://rpm5.org CVS Sources Repository[EMAIL PROTECTED] __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: [CVS] RPM: rpm/db3/ configure
Ralf S. Engelschall wrote: On Wed, Jun 27, 2007, Mark Hatle wrote: This actually points out a bug I found the other day, but haven't been able to figure out how to solve. Using echo $* in this way causes variable or argument expansions that include a space to fail. We have a few cases where we need to pass in CFLAGS w/ multiple elements or CC='distcc ccache foo-gcc -te500v1'.. which causes a failure to occur. I replaced the args bit with a simple $@ in the args list to work around the issue.. but I don't know of a way of modifying the arguments in the shell and passing them properly quoted. Well, just $@ doesn't fully help. It quotes the stuff just for the echo(1) call itself, but after echo(1) outputs the stuff to stdout the quotes will be gone anyway. There is only one possibility in Bourne-Shell: to use a for loop to iterate over the arguments and then re-construct the stuff in a quoted way. Sorry I wasn't clear.. I removed the args processing, and changed out $args in the configure call w/ $@ and at least in my case it worked properly. I'm not exactly sure what we are protecting w/ the sed's in the args. I think the alternative is to copy what configure does and properly quote each of the arguments. There is probably some code we can borrow from configure itself to do this. --Mark __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: [CVS] RPM: rpm/ CHANGES configure.ac rpm/rpmdb/ Makefile.am
On Jun 27, 2007, at 6:44 PM, Ralf S. Engelschall wrote: RPM Package Manager, CVS Repository http://rpm5.org/cvs/ __ __ Server: rpm5.org Name: Ralf S. Engelschall Root: /v/rpm/cvs Email: [EMAIL PROTECTED] Module: rpm Date: 28-Jun-2007 00:44:56 Branch: HEAD Handle: 2007062723445500 Modified files: rpm CHANGES configure.ac rpm/rpmdb Makefile.am Log: Simplify internal Berkeley-DB handling in rpmdb/ even further and as a wished side-effect allow (at least in an clearly unsupported way to make Jeff not panic ;-) the use of --with-db=external. It's not a question of panic, but rather support and feature regression. Let's say someone wants to us db-4.1.25. rpm compiles fine. the default db configuration breaks because not tuned to that version, i.e. no stale lock removal. Then there is the select poll in db-4.1.25, which is an order of magnitude slower with a 5000 item test db than either the previous or the next verrsion of Berkeley DB. I *really* don't want to diagnose these problems again and again and again because lusers want choice until it hurts. 73 de Jeff __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
