Re: [RFC] OpenLDAP automatic upgrade
Hi Quanah, On Mon, Mar 21, 2005 at 05:58:01PM -0800, Quanah Gibson-Mount wrote: You can find the patch for adding -q to slapadd on OpenLDAP 2.2 here: http://www.stanford.edu/services/directory/openldap/configuration/openldap-build-42.html Great, thanks! Applied it to the subversion repository. I've been reading back over my notes on the problems OpenLDAP has with BDB 4.3, and I believe that this patch will resolve those issues as well (as long as it is used to load a DB, rather than without the -q flag). The main issue is that the way BDB 4.3 creates transaction logs in the quickload scenario is to use setflags DB_LOG_INMEMORY Okay. Anyway, I guess I'll revert to bdb 4.2 if you made bad experiences with 4.3. As long as OpenLDAP does not need any features from 4.3 why would we want to use it anyway? I would note that it would likely be good for Debian to still include a DB_CONFIG file in the database directory it creates with its OpenLDAP distribution, as the default settings from Sleepycat are entirely too small. Here are some examples. All examples use: 10k entry LDIF file 23 attributes indexed BDB 4.2.52 database bdb as the slapd.conf database 1) DB_CONFIG file with a 384MB cache, and normal slapadd options. ldap-linux0:/db/DBs# time slapadd -l 10k.ldif 30.760u 1.800s 0:36.89 88.2%0+0k 0+0io 13351pf+0w Point taken :) I'll submit this as a bug to not forget about it. I think about using at least 1MB for the cache (stop laughing :-) and at most 10% of the main memory of the system. After all there are still people running slapd on systems with 16MB of memory (our students' hostel server is still running such a system...) Greetings Torsten signature.asc Description: Digital signature
Re: [RFC] OpenLDAP automatic upgrade
Hi Quanah, On Thu, Mar 17, 2005 at 03:39:09PM -0800, Quanah Gibson-Mount wrote: Is there a way to enforce this without editing DB_CONFIG? I'd rather set an environment variable or something like that. Writing that into DB_CONFIG in the maintainer scripts always poses the risk that it'll stay. Well This will be available in OpenLDAP 2.3 via the -q option to slapadd. I have my own backported patch to OpenLDAP 2.2 that enables the -q option. Care to share that patch? :) Greetings Torsten signature.asc Description: Digital signature
Re: [RFC] OpenLDAP automatic upgrade
Torsten Landschoff [EMAIL PROTECTED] writes: Hi Quanah, On Thu, Mar 17, 2005 at 03:39:09PM -0800, Quanah Gibson-Mount wrote: Is there a way to enforce this without editing DB_CONFIG? I'd rather set an environment variable or something like that. Writing that into DB_CONFIG in the maintainer scripts always poses the risk that it'll stay. Well This will be available in OpenLDAP 2.3 via the -q option to slapadd. I have my own backported patch to OpenLDAP 2.2 that enables the -q option. Care to share that patch? :) Torsten, You can find the patch for adding -q to slapadd on OpenLDAP 2.2 here: http://www.stanford.edu/services/directory/openldap/configuration/openldap-build-42.html I've been reading back over my notes on the problems OpenLDAP has with BDB 4.3, and I believe that this patch will resolve those issues as well (as long as it is used to load a DB, rather than without the -q flag). The main issue is that the way BDB 4.3 creates transaction logs in the quickload scenario is to use setflags DB_LOG_INMEMORY That particular function when combined with how OpenLDAP loads data often results in the server running out of memory, and the entire environment becoming corrupted. The -q patch completely disables the creation of any logs by either BDB 4.2 or BDB 4.3, and significantly decreases the amount of time it takes to load a database. I would note that it would likely be good for Debian to still include a DB_CONFIG file in the database directory it creates with its OpenLDAP distribution, as the default settings from Sleepycat are entirely too small. Here are some examples. All examples use: 10k entry LDIF file 23 attributes indexed BDB 4.2.52 database bdb as the slapd.conf database 1) DB_CONFIG file with a 384MB cache, and normal slapadd options. ldap-linux0:/db/DBs# time slapadd -l 10k.ldif 30.760u 1.800s 0:36.89 88.2%0+0k 0+0io 13351pf+0w 2) DB_CONFIG file with a 384MB cache, and disabling logging via the BDB 4.2.52 flags. ldap-linux0:/db/DBs# time slapadd -l 10k.ldif 26.790u 0.410s 0:27.35 99.4%0+0k 0+0io 13318pf+0w 3) DB_CONFIG file with a 384MB cache and using the -q option to slapadd. ldap-linux0:/db/DBs# time slapadd -q -l 10k.ldif 17.270u 0.450s 0:17.88 99.1%0+0k 0+0io 13221pf+0w 4) no DB_CONFIG file and normal slapadd options. Note that this is Debian's current default method. ldap-linux0:/db/DBs# time slapadd -l 10k.ldif 38.920u 26.830s 13:30.39 8.1% 0+0k 0+0io 732pf+0w 5) no DB_CONFIG file and using the -q option to slapadd. ldap-linux0:/db/DBs# time slapadd -q -l 10k.ldif 25.470u 21.070s 0:50.10 92.8% 0+0k 0+0io 627pf+0w --Quanah -- Quanah Gibson-Mount Principal Software Developer ITSS/Shared Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
On Thursday 17 March 2005 02:59, Quanah Gibson-Mount wrote: That's for sure but I want to be able to do automatic upgrades for the simple cases. And at least help the admin by dumping the directory before starting the upgrade and taking care of the old database files in case he decides to downgrade again later :) I don't think there is a simple case. ;) I serve ~1500 Users via nss-ldap and friends from OpenLDAP only using additional indices and schema-enhancements. Then again I also regenerate the whole LDAP tree every night from the enterprise RDBMS making me too no big candidate for automatic upgrades. Regards, David -- - hallo... wie gehts heute? - *hust* gut *rotz* *keuch* - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15
Re: [RFC] OpenLDAP automatic upgrade
Hi Quanah, On Wed, Mar 16, 2005 at 05:59:03PM -0800, Quanah Gibson-Mount wrote: The patches for BDB 4.2.52 are freely available from Sleepycat. They are required to be in place if you want a stable BDB 4.2.52 distribution. I would be very surprised if the package maintainer hadn't already included the patches in their build. I also post links to the patches from my Make we wonder why Sleepycat does not release 4.2.53 with those patches included. website on building OpenLDAP: http://www.stanford.edu/services/directory/openldap/configuration/ specifically in this case: http://www.stanford.edu/services/directory/openldap/configuration/bdb-build-42.html Thanks, I was pointed to that site by another developer already. I just did not like applying patches without really knowing what they are needed for. You can ignore the custom patch about transactions, as that is not an official sleepycat patch, but one written by Howard Chu, one of the primary OpenLDAP developers. What is that patch for anyway? Seems like it allows slapd to open a transaction and never commit it from first sight. Why would that be needed? As for BDB 4.3: You cannot use BDB 4.3 to load databases past a few million entries into OpenLDAP. The way BDB handles logs changed enormously between BDB 4.2 and BDB 4.3, and its log management is not stable, often running out of space. In addition, numerous users have written the OpenLDAP list complaining of issues they've hit using BDB 4.3 with OpenLDAP 2.2. The solution was for them to move back to BDB 4.2.52+patches. This became such a problem that with the OpenLDAP 2.2.24 release (done today), the README file was updated to say: SLAPD: BDB backend requires (latest) Sleepycat Berkeley DB 4.2 Good to know. I'll revert the Debian package to 4.2. Surely. For simple installations it might work without that though and not every slapd package is installed to run the Stanford directory server. :) I doubt that it will work for even simple installations. The very syntax of many of the ACL declarations were changed. I had to go through and update every single ACL in my ACL file for it to work right. ;) There are some problems I found during upgrading an example installation which are automatically fixed. Those are mostly just for the Debian default installation though. That's what I am trying to do: Make it work for people who just did apt-get install slapd and never cared much about that LDAP directory. The two flags I am thinking of for BDB 4.2 are: # Just use these settings when doing slapadd... #set_flags DB_TXN_NOSYNC #set_flags DB_TXN_NOT_DURABLE They turn off checks that aren't necessary when performing a slapadd. Is there a way to enforce this without editing DB_CONFIG? I'd rather set an environment variable or something like that. Writing that into DB_CONFIG in the maintainer scripts always poses the risk that it'll stay. I also advise reading: http://www.stanford.edu/services/directory/openldap/configuration/bdb-config-42.html http://www.openldap.org/faq/data/cache/1075.html Done, thanks. Nothing really new there though. If you really want to have this conversation, I suggest talking to Howard Chu (Howard Chu [EMAIL PROTECTED]). He's the author of back-bdb and back-hdb (The OL development team tends to refer to back-hdb as Howard's DB. :P). :) Perhaps I should do that but I know what the answer might be: Go ahead and implement it ;-) Which is what I'd answer from his point of view as well. And currently I don't care that much. If it is slow, so be it. Important is that it is working. I'll take some of the information you linked to into the README.Debian of slapd in the hopes that it is read by some people. 4) If someone is using ldbm as their backend database, I'd throw a warning noting that bdb/hdb are the preferred backends of OpenLDAP. ldbm is rife with issues such as not catching data inconsistencies, leading to poor directory performance and potential data loss. Where is the difference to bdb? ;-) [...] As an individual who dealt with an ldbm based directory, I can attest to the superiority of back-bdb. That was supposed to be a joke about the stability of bdb. Looking at the reports I was getting about bdb problems it seems like bdb has some issues. From the feedback I get I guess that most are fixed in bdb. Okay. 2.2.24 was released. You might examine the changes file to see if you want to use it, or pull in patches that it incorporated. Have to check... Would be a good time to switch back to bdb 4.2 together with switching to 2.2.24 of OpenLDAP. That's for sure but I want to be able to do automatic upgrades for the simple cases. And at least help the admin by dumping the directory before starting the upgrade and taking care of the old database files in case he decides to downgrade again later :) I don't think there is a simple case. ;) Oh yes there is. I
Re: [RFC] OpenLDAP automatic upgrade
On Thu, Mar 17, 2005 at 12:10:23AM -0300, Henrique de Moraes Holschuh wrote: In short, I cannot find a single reason to run OpenLDAP against BDB 4.3, and even the current OpenLDAP release notes that BDB 4.2 is required. I can find many reasons to not use BDB 4.3. Not good. That might mean a lot of trouble to get 4.3 out of sarge, and revert all packages back to 4.2 :( As far as OpenLDAP is concerned, 2.2.23 with bdb 4.3 did not yet enter even sid. The following packages are found by apt-cache showpkg: Reverse Depends: slapd,libdb4.3 xkbsel,libdb4.3 webdruid,libdb4.3 python2.3-librdf,libdb4.3 python2.2-librdf,libdb4.3 oops,libdb4.3 librdf0,libdb4.3 librdf-ruby,libdb4.3 librdf-perl,libdb4.3 libdb4.3-dev,libdb4.3 4.3.27-2 libberkeleydb-perl,libdb4.3 gnunet,libdb4.3 db4.3-util,libdb4.3 bogofilter,libdb4.3 slapd,libdb4.3 Should we notify the maintainers to better go back to 4.2 for sarge? Greetings Torsten signature.asc Description: Digital signature
Re: [RFC] OpenLDAP automatic upgrade
On Fri, 18 Mar 2005, Torsten Landschoff wrote: Should we notify the maintainers to better go back to 4.2 for sarge? I think so. Jumping into a new BDB version like it was done for 4.3 is *always* foolhardy at best, IMHO. But it would be a very good idea to track down some other opinions about BDB 4.3 first, as people will not like the idea of downgrading BDB very much (db_upgrade cannot help you with that...) -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
Torsten Landschoff [EMAIL PROTECTED] writes: Hi Quanah, On Wed, Mar 16, 2005 at 05:59:03PM -0800, Quanah Gibson-Mount wrote: The patches for BDB 4.2.52 are freely available from Sleepycat. They are required to be in place if you want a stable BDB 4.2.52 distribution. I would be very surprised if the package maintainer hadn't already included the patches in their build. I also post links to the patches from my Make we wonder why Sleepycat does not release 4.2.53 with those patches included. I've wondered that myself. ;) You can ignore the custom patch about transactions, as that is not an official sleepycat patch, but one written by Howard Chu, one of the primary OpenLDAP developers. What is that patch for anyway? Seems like it allows slapd to open a transaction and never commit it from first sight. Why would that be needed? It fixes a case where BDB log files never get closed because of uncommitted transactions, IIRC. I have a link to the entire discussion about it from my web page if you want to read all about it. ;) It basically implements in BDB 4.2 the one useful feature of BDB 4.3 as far as OpenLDAP is concerned. Surely. For simple installations it might work without that though and not every slapd package is installed to run the Stanford directory server. :) I doubt that it will work for even simple installations. The very syntax of many of the ACL declarations were changed. I had to go through and update every single ACL in my ACL file for it to work right. ;) There are some problems I found during upgrading an example installation which are automatically fixed. Those are mostly just for the Debian default installation though. That's what I am trying to do: Make it work for people who just did apt-get install slapd and never cared much about that LDAP directory. Okay. I've never used the debian default package myself, so maybe that is that simple. :) The two flags I am thinking of for BDB 4.2 are: # Just use these settings when doing slapadd... #set_flags DB_TXN_NOSYNC #set_flags DB_TXN_NOT_DURABLE They turn off checks that aren't necessary when performing a slapadd. Is there a way to enforce this without editing DB_CONFIG? I'd rather set an environment variable or something like that. Writing that into DB_CONFIG in the maintainer scripts always poses the risk that it'll stay. Well This will be available in OpenLDAP 2.3 via the -q option to slapadd. I have my own backported patch to OpenLDAP 2.2 that enables the -q option. If you really want to have this conversation, I suggest talking to Howard Chu (Howard Chu [EMAIL PROTECTED]). He's the author of back-bdb and back-hdb (The OL development team tends to refer to back-hdb as Howard's DB. :P). ) Perhaps I should do that but I know what the answer might be: Go ahead and implement it ;-) Which is what I'd answer from his point of view as well. And currently I don't care that much. If it is slow, so be it. Important is that it is working. I'll take some of the information you linked to into the README.Debian of slapd in the hopes that it is read by some people. I asked Howard about this actually... Another developer is examining doing this at this time. It would be in OpenLDAP 2.3 at the earliest, however (and I'm guessing 2.4, since I haven't seen any commits over it). 4) If someone is using ldbm as their backend database, I'd throw a warning noting that bdb/hdb are the preferred backends of OpenLDAP. ldbm is rife with issues such as not catching data inconsistencies, leading to poor directory performance and potential data loss. Where is the difference to bdb? ;-) [...] As an individual who dealt with an ldbm based directory, I can attest to the superiority of back-bdb. That was supposed to be a joke about the stability of bdb. Looking at the reports I was getting about bdb problems it seems like bdb has some issues. From the feedback I get I guess that most are fixed in bdb. Did you use bdb here a few times when you meant to use ldbm? ;) Anyhow, with OpenLDAP 2.2 + BDB 4.2.52+patches, my BDB database has been rock solid. Issues I have had arose from bugs in OL not BDB. ;) Oh yes there is. I know quite some directory servers on our university campus running Debian slapd as shipped with no special config. Just putting 500 users into it and using it as NIS replacement. That's something even OpenLDAP 2.0 was doing quite well... I'll take your word for it. :) --Quanah -- Quanah Gibson-Mount Principal Software Developer ITSS/Shared Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
Should we notify the maintainers to better go back to 4.2 for sarge? Don't bother notifying me; I won't be switching anything back to 4.2. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
On Thu, 17 Mar 2005, Clint Adams wrote: Should we notify the maintainers to better go back to 4.2 for sarge? Don't bother notifying me; I won't be switching anything back to 4.2. Why? (technical reasons, please). Not that I am assuming there is enough evidence to downgrade anything but OpenLDAP just yet, but your reply seems to imply that even if there were, you would still not downgrade. Of course, I don't mean downgrading the libdb4.3 packages :-) Those would just get a grave bug until the issue is fixed (assuming there is one, which is not clear at this point). -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
Why? (technical reasons, please). Not that I am assuming there is enough evidence to downgrade anything but OpenLDAP just yet, but your reply seems to imply that even if there were, you would still not downgrade. If there were anything besides FUD, I'd consider it on its own merits, but all I've seen thus far is an anecdote that OpenLDAP has trouble with some version of db4.3 on some platform because of some undescribed flaw related to the log format change. There does not appear to be a report in the Debian BTS about this problem. So, given that I don't see any reason to expect problems from db4.3, and that it would be painful for sarge and sid users to switch back to db4.2, I don't intend to do so. Now, as far as pestering other maintainers goes, I don't believe there's a point there either. Most of the packages currently built against libdb4.3 don't use transactional environments, and thus cannot be bitten by the txn log problem mentioned by Quanah Gibson-Mount. If there are any real problems with software built with Debian's db4.3 packages (which are built quite differently than Fedora's, for example), they should be reported so they can be fixed. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
On Fri, 18 Mar 2005, Henrique de Moraes Holschuh wrote: Of course, I don't mean downgrading the libdb4.3 packages :-) Those would just get a grave bug until the issue is fixed (assuming there is one, which is not clear at this point). Before even bothering to continue this thread, whoever claims that there is a problem with libdb4.3 needs to demonstrate the problem and file a bug against the appropriate package. According to http://bugs.debian.org/cgi-bin/pkgreport.cgi?src=db4.3 there are no open bugs that even remotely resemble the problems being reported. Otherwise all we're doing is windmill tilting. Don Armstrong -- There is no mechanical problem so difficult that it cannot be solved by brute strength and ignorance. -- William's Law http://www.donarmstrong.com http://rzlab.ucr.edu -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
On Thu, 17 Mar 2005, Clint Adams wrote: some version of db4.3 on some platform because of some undescribed flaw related to the log format change. There does not appear to be a report in the Debian BTS about this problem. Hmm... my experience with BDB 4.x tells me we should be quite a bit paranoid with it, it is a wonderful piece of software, but it takes a while to shake out all the bugs, and so far that while was more time than what it took for upstream to change to the next version. Only now I would trust BDB 4.2 with any mission critical data... but then, I am the one which still builds Cyrus 2.1 against BDB 3.2 for stability (Cyrus 2.2 will be built against BDB 4.2). On a tangent, why do we still have BDB 4.1 on Debian? Isn't it not exactly safe on SMP and SMT machines? Or were all bugs fixed in 4.2 also fixed there? Now, as far as pestering other maintainers goes, I don't believe there's a point there either. Most of the packages currently built against Not yet, that's for sure. libdb4.3 don't use transactional environments, and thus cannot be bitten by the txn log problem mentioned by Quanah Gibson-Mount. Agreed. If there are any real problems with software built with Debian's db4.3 packages (which are built quite differently than Fedora's, for example), they should be reported so they can be fixed. I also agree with you in this. I will take that as a prove to me that there is a bug, and we go from there reply. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
Only now I would trust BDB 4.2 with any mission critical data... but then, I am the one which still builds Cyrus 2.1 against BDB 3.2 for stability (Cyrus 2.2 will be built against BDB 4.2). IIRC, BDB 3.3 addresses very serious problems in 3.2, but we can't have 3.3 in Debian without a painful libdb3 transition. On a tangent, why do we still have BDB 4.1 on Debian? Isn't it not exactly safe on SMP and SMT machines? Or were all bugs fixed in 4.2 also fixed there? As soon as packages stop depending on 4.1, it is likely to be removed altogether, just as 4.0 was. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
Clint Adams [EMAIL PROTECTED] writes: If there were anything besides FUD, I'd consider it on its own merits, but all I've seen thus far is an anecdote that OpenLDAP has trouble with some version of db4.3 on some platform because of some undescribed flaw related to the log format change. There does not appear to be a report in the Debian BTS about this problem. You should probably bear in mind that the problem was judged by the OpenLDAP maintainers to be sufficiently severe that they removed db4.3 from their supported db version list in the latest release of OpenLDAP. That's a bit more than FUD, although I agree that it's not specifics. I agree that it's unclear at the moment whether the problem is unique to OpenLDAP or whether other applications would encounter the same problem. -- Russ Allbery ([EMAIL PROTECTED]) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
Clint Adams [EMAIL PROTECTED] writes: Why? (technical reasons, please). Not that I am assuming there is enough evidence to downgrade anything but OpenLDAP just yet, but your reply seems to imply that even if there were, you would still not downgrade. If there were anything besides FUD, I'd consider it on its own merits, but all I've seen thus far is an anecdote that OpenLDAP has trouble with some version of db4.3 on some platform because of some undescribed flaw related to the log format change. There does not appear to be a report in the Debian BTS about this problem. The problem has nothing to do with a log format change. It has to do with the way BDB 4.3 handles logs, period. And it isn't any one platform, I've encountered it on multiple linux 2.4 kernel based platforms, Solaris 8, and Solaris 9. The most common error that happens is the BDB log environment suddenly claims it is out of memory, and it is then impossible to run db_archive or db_recover. Applications that use the DB_INMEMORY_LOGS flags for BDB 4.3 are especially prone to this issue. So, given that I don't see any reason to expect problems from db4.3, and that it would be painful for sarge and sid users to switch back to db4.2, I don't intend to do so. Now, as far as pestering other maintainers goes, I don't believe there's a point there either. Most of the packages currently built against libdb4.3 don't use transactional environments, and thus cannot be bitten by the txn log problem mentioned by Quanah Gibson-Mount. I never said there was a transaction log problem. I said that BDB 4.2 doesn't support the transaction methods that BDB 4.3 was, which is a deficiency in BDB 4.2, not 4.3, that can be mended by a particular patch to BDB 4.2, but I don't recommend doing that for the general case, because whatever application is being compiled against BDB 4.2 in that case will also likely need patching. If there are any real problems with software built with Debian's db4.3 packages (which are built quite differently than Fedora's, for example), they should be reported so they can be fixed. I suggest you read back over what I wrote in previous posts more carefully, because you seemed to miss most of it. --Quanah -- Quanah Gibson-Mount Principal Software Developer ITSS/Shared Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
On Tue, Mar 15, 2005 at 05:04:26PM -0800, Quanah Gibson-Mount wrote: The first. Basically upstream changes the database format quite often. I am even not entirely sure if the database format stays compatible in the 2.1 or 2.2 line but I'd expect it to. The 2.2.23 Debian packages uses libdb4.3 instead of libdb4.2 as used in 2.1.x so the reload has to be done in any case. Just to be very clear on this again. You do *not* want to run OpenLDAP against BDB 4.3. Releasing Debian with its OpenLDAP compiled against BDB 4.3 would be a serious mistake. And this is why? So far 2.2.23 with libdb4.3 seems to be much more stable than anything before. Greetings Torsten signature.asc Description: Digital signature
Re: [RFC] OpenLDAP automatic upgrade
Hi Quanah, On Tue, Mar 15, 2005 at 01:07:29PM -0800, Quanah Gibson-Mount wrote: I currently maintain Stanford University's directory service, which is based on OpenLDAP. I also am a member of the OpenLDAP core team, and I hold down another job with Symas Corporation doing a variety of tasks, but most relevantly benchmarking their OpenLDAP based product and teaching classes on running and configuring OpenLDAP based directory services. Whew! I have some comments on your upgrade plans listed here... ;) Great, fire! :) Going from OpenLDAP 2.0 to 2.1 was non-trivial. Going from 2.0 to 2.2 is very complex. Going from 2.1 to 2.2 is also non-trivial. Talk about stating the obvious ;) Some issues to consider for the 2.1 to 2.2 process: 1) OpenLDAP 2.1 generally used BDB 4.1. OpenLDAP 2.2 should only be used with BDB 4.2.52+patches when using bdb or hdb as the backend (Ignore the documentation with OpenLDAP 2.2.23 stating it requires BDB 4.3, that statement is incorrect). Yes, I accidently built it with BDB 4.2 and it worked. But - what is the problem with 4.3? And what patches will be needed for 4.2.52 - it's not like I can go and change the libdb4.2 packages, I'll have to ask the maintainer for that favour. 2) Several changes were made to how ACL's were handled between OpenLDAP 2.1 and OpenLDAP 2.2. You cannot simply upgrade the database and expect everything to work. The ACL's must also be reviewed and correctly updated. Surely. For simple installations it might work without that though and not every slapd package is installed to run the Stanford directory server. :) 3) OpenLDAP bdb/hdb based databases are highly dependent on a well configured DB_CONFIG file for the particular database in question. Simply doing a slapcat/slapadd is really not sufficient for ensuring that the resulting directory server will run well. A poorly configured DB_CONFIG file can lead to the slapadd process taking hours or days. In addition, there are two flags that should be set in the DB_CONFIG file before the slapadd takes place, that should then be commented out once the add is completed. Erm, the upgrading process is already really complex, and I fear I don't feel like implementing this in maintainer scripts. Which options are you thinking of, BTW? And the documentation for DB_CONFIG is really lacking as well, let's hope somebody wakes up the sleepy cats ;) I wonder why the OpenLDAP server can not tune most of the BDB parameters itself during runtime. It has by far more information at its hands than the Debian maintainer scripts have. 4) If someone is using ldbm as their backend database, I'd throw a warning noting that bdb/hdb are the preferred backends of OpenLDAP. ldbm is rife with issues such as not catching data inconsistencies, leading to poor directory performance and potential data loss. Where is the difference to bdb? ;-) 5) I don't know what version of OpenLDAP 2.2 that debian is considering for inclusion, but I wouldn't go with anything less than OpenLDAP 2.2.23 based on my experiences. That's what I am working on. Essentially, the process of upgrading a directory service to OpenLDAP 2.2 is generally something that should be done by the person who runs the service. There are a host of issues that must be addressed to properly upgrade. That's for sure but I want to be able to do automatic upgrades for the simple cases. And at least help the admin by dumping the directory before starting the upgrade and taking care of the old database files in case he decides to downgrade again later :) Thanks for your comments Torsten signature.asc Description: Digital signature
Re: [RFC] OpenLDAP automatic upgrade
* Quanah Gibson-Mount wrote: [...] You do *not* want to run OpenLDAP against BDB 4.3. Releasing Debian with its OpenLDAP compiled against BDB 4.3 would be a serious mistake. You forgot to explain _why_ OpenLDAP compiled against BDB 4.3 should be a serious mistake. Norbert -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
Torsten Landschoff [EMAIL PROTECTED] writes: Some issues to consider for the 2.1 to 2.2 process: 1) OpenLDAP 2.1 generally used BDB 4.1. OpenLDAP 2.2 should only be used with BDB 4.2.52+patches when using bdb or hdb as the backend (Ignore the documentation with OpenLDAP 2.2.23 stating it requires BDB 4.3, that statement is incorrect). Yes, I accidently built it with BDB 4.2 and it worked. But - what is the problem with 4.3? And what patches will be needed for 4.2.52 - it's not like I can go and change the libdb4.2 packages, I'll have to ask the maintainer for that favour. The patches for BDB 4.2.52 are freely available from Sleepycat. They are required to be in place if you want a stable BDB 4.2.52 distribution. I would be very surprised if the package maintainer hadn't already included the patches in their build. I also post links to the patches from my website on building OpenLDAP: http://www.stanford.edu/services/directory/openldap/configuration/ specifically in this case: http://www.stanford.edu/services/directory/openldap/configuration/bdb-build-42.html You can ignore the custom patch about transactions, as that is not an official sleepycat patch, but one written by Howard Chu, one of the primary OpenLDAP developers. As for BDB 4.3: You cannot use BDB 4.3 to load databases past a few million entries into OpenLDAP. The way BDB handles logs changed enormously between BDB 4.2 and BDB 4.3, and its log management is not stable, often running out of space. In addition, numerous users have written the OpenLDAP list complaining of issues they've hit using BDB 4.3 with OpenLDAP 2.2. The solution was for them to move back to BDB 4.2.52+patches. This became such a problem that with the OpenLDAP 2.2.24 release (done today), the README file was updated to say: SLAPD: BDB backend requires (latest) Sleepycat Berkeley DB 4.2 I've been running OpenLDAP 2.2 with BDB 4.2.52+patches since OpenLDAP 2.2.6 was released, and any issues I've had were with bugs in OpenLDAP, not with BDB. I also ran benchmarks against BDB 4.3 based OpenLDAP 2.2 servers, and they performed no better (and slightly worse) than BDB 4.2 based OpenLDAP 2.2 servers. In short, I cannot find a single reason to run OpenLDAP against BDB 4.3, and even the current OpenLDAP release notes that BDB 4.2 is required. I can find many reasons to not use BDB 4.3. 2) Several changes were made to how ACL's were handled between OpenLDAP 2.1 and OpenLDAP 2.2. You cannot simply upgrade the database and expect everything to work. The ACL's must also be reviewed and correctly updated. Surely. For simple installations it might work without that though and not every slapd package is installed to run the Stanford directory server. :) I doubt that it will work for even simple installations. The very syntax of many of the ACL declarations were changed. I had to go through and update every single ACL in my ACL file for it to work right. ;) 3) OpenLDAP bdb/hdb based databases are highly dependent on a well configured DB_CONFIG file for the particular database in question. Simply doing a slapcat/slapadd is really not sufficient for ensuring that the resulting directory server will run well. A poorly configured DB_CONFIG file can lead to the slapadd process taking hours or days. In addition, there are two flags that should be set in the DB_CONFIG file before the slapadd takes place, that should then be commented out once the add is completed. Erm, the upgrading process is already really complex, and I fear I don't feel like implementing this in maintainer scripts. Which options are you thinking of, BTW? And the documentation for DB_CONFIG is really lacking as well, let's hope somebody wakes up the sleepy cats ;) The two flags I am thinking of for BDB 4.2 are: # Just use these settings when doing slapadd... #set_flags DB_TXN_NOSYNC #set_flags DB_TXN_NOT_DURABLE They turn off checks that aren't necessary when performing a slapadd. I also advise reading: http://www.stanford.edu/services/directory/openldap/configuration/bdb-config-42.html http://www.openldap.org/faq/data/cache/1075.html I wonder why the OpenLDAP server can not tune most of the BDB parameters itself during runtime. It has by far more information at its hands than the Debian maintainer scripts have. If you really want to have this conversation, I suggest talking to Howard Chu (Howard Chu [EMAIL PROTECTED]). He's the author of back-bdb and back-hdb (The OL development team tends to refer to back-hdb as Howard's DB. :P). 4) If someone is using ldbm as their backend database, I'd throw a warning noting that bdb/hdb are the preferred backends of OpenLDAP. ldbm is rife with issues such as not catching data inconsistencies, leading to poor directory performance and potential data loss. Where is the difference to bdb? ;-) http://www.openldap.org/faq/data/cache/756.html Pay close attention to: back-bdb uses BDB's most advanced
Re: [RFC] OpenLDAP automatic upgrade
On Wed, 16 Mar 2005, Quanah Gibson-Mount wrote: The patches for BDB 4.2.52 are freely available from Sleepycat. They are required to be in place if you want a stable BDB 4.2.52 distribution. I would be very surprised if the package maintainer hadn't already included the patches in their build. I also post links to the patches from my Yes, they are included. In fact, I would expect Debian BDB 4.2 to be much more stable than whatever is available from Sleepycat, as a rule of thumb. The dire days of BDB 4.2 royally screwing over your database in any non-joke box are gone, fortunately. As for BDB 4.3, I have no idea. As for BDB 4.3: You cannot use BDB 4.3 to load databases past a few million entries into OpenLDAP. The way BDB handles logs changed enormously between BDB 4.2 and BDB 4.3, and its log management is not stable, often running out of space. In addition, numerous users have written the OpenLDAP list complaining of issues they've hit using BDB 4.3 with OpenLDAP 2.2. The solution was for them to move back to BDB 4.2.52+patches. This became I would expect that to cause trouble with just about everything that uses huge indexes and databases with BDB 4.3. Or does it show up only in OpenLDAP usage patterns (OpenLDAP is known to find all lurking bugs in BDB that others never hit :-) ). In short, I cannot find a single reason to run OpenLDAP against BDB 4.3, and even the current OpenLDAP release notes that BDB 4.2 is required. I can find many reasons to not use BDB 4.3. Not good. That might mean a lot of trouble to get 4.3 out of sarge, and revert all packages back to 4.2 :( -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
Henrique de Moraes Holschuh [EMAIL PROTECTED] writes: On Wed, 16 Mar 2005, Quanah Gibson-Mount wrote: The patches for BDB 4.2.52 are freely available from Sleepycat. They are required to be in place if you want a stable BDB 4.2.52 distribution. I would be very surprised if the package maintainer hadn't already included the patches in their build. I also post links to the patches from my Yes, they are included. In fact, I would expect Debian BDB 4.2 to be much more stable than whatever is available from Sleepycat, as a rule of thumb. Good good. :) As for BDB 4.3: You cannot use BDB 4.3 to load databases past a few million entries into OpenLDAP. The way BDB handles logs changed enormously between BDB 4.2 and BDB 4.3, and its log management is not stable, often running out of space. In addition, numerous users have written the OpenLDAP list complaining of issues they've hit using BDB 4.3 with OpenLDAP 2.2. The solution was for them to move back to BDB 4.2.52+patches. This became I would expect that to cause trouble with just about everything that uses huge indexes and databases with BDB 4.3. Or does it show up only in OpenLDAP usage patterns (OpenLDAP is known to find all lurking bugs in BDB that others never hit :-) ). Well, I only use BDB with OpenLDAP, so I can't say one way or the other. ;) In short, I cannot find a single reason to run OpenLDAP against BDB 4.3, and even the current OpenLDAP release notes that BDB 4.2 is required. I can find many reasons to not use BDB 4.3. Not good. That might mean a lot of trouble to get 4.3 out of sarge, and revert all packages back to 4.2 :( Well, I'd say at least for OpenLDAP 2.2, it really needs to be done if a Debian wants to release a stable OpenLDAP setup. ;) --Quanah -- Quanah Gibson-Mount Principal Software Developer ITSS/Shared Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
On Tue, Mar 15, 2005 at 01:36:17AM +0100, Torsten Landschoff wrote: As you might have noticed or not we are working on getting OpenLDAP 2.2 into unstable. The packages are mostly working fine (as available in experimental) but what is missing is a really tested upgrade path from OpenLDAP 2.0 (in stable) and 2.1 (in testing, unstable). Ambitious target! I upgraded my employer's LDAP servers from 2.1 to 2.2 last week. They are running on Debian machines, but not using the Debian packages. Here are some points which you might find useful: * Fix/check the DB_CONFIG file if you're using BDB. Having a suboptimal DB_CONFIG will really really slow down the slapadd. * There are syntax changes in slapd.conf, e.g. group.regex becomes group.expand, and there are new facilities which are potentially desirable to add, e.g. checkpoints. Also ACL parsing is stricter. * Strip out the entryCSN attributes from the ldif file and let slapadd recreate them in the 2.2 format. That's necessary for using syncrepl in the future and it might as well be done while you're tearing the database to pieces. * The monitor back-end behaves a little differently in 2.2 to previous versions, so monitoring scripts need tweaking to continue to work. All in all, it's not a difficult upgrade, but there a lot of little things to get right. Dave pgphARIFBcQ4H.pgp Description: PGP signature
Re: [RFC] OpenLDAP automatic upgrade
sean finney [EMAIL PROTECTED] writes: On Tue, Mar 15, 2005 at 01:36:17AM +0100, Torsten Landschoff wrote: a) the preinst checks if the database format has changed between the old version and the version that we are upgrading to is this an underlying database format change, or simply stricter schema checks? if it is a change to the db format, will the new server work with the old format (even if less optimally)? if so it might make a better quality package to leave the data in the old format and provide instructions to the admin (who will know more than you about the directory server) for how to get the new optimised format. OpenLDAP 2.1 used hash trees with the BDB backend. OpenLDAP 2.2 uses btree trees with the BDB backend. The formats are not compatible. --Quanah -- Quanah Gibson-Mount Principal Software Developer ITSS/Shared Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
On Tue, Mar 15, 2005 at 02:31:01PM +, Dave Holland wrote: On Tue, Mar 15, 2005 at 01:36:17AM +0100, Torsten Landschoff wrote: As you might have noticed or not we are working on getting OpenLDAP 2.2 into unstable. The packages are mostly working fine (as available in experimental) but what is missing is a really tested upgrade path from OpenLDAP 2.0 (in stable) and 2.1 (in testing, unstable). Ambitious target! I upgraded my employer's LDAP servers from 2.1 to 2.2 last week. They are running on Debian machines, but not using the Debian packages. Here are some points which you might find useful: [... some helpfull slapd.conf issues deleted ...] All in all, it's not a difficult upgrade, but there a lot of little things to get right. the ldif format was not a problem? was the syntax parsed correctly from 2.1 to 2.2? -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
Andreas Schuldei [EMAIL PROTECTED] writes: the ldif format was not a problem? was the syntax parsed correctly from 2.1 to 2.2? LDIF is rfc described... there shouldn't be any issue using the LDIF output. The main issue is possibly stripping the entryCSN for the use of syncrepl. However, syncrepl was poorly implemented in OpenLDAP 2.2, and i would advise against its use. OpenLDAP 2.3 has a completely rewritten syncrepl implementation that is stable. However, OpenLDAP 2.3 is still in the beta stages. --Quanah -- Quanah Gibson-Mount Principal Software Developer ITSS/Shared Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
Hi Sean, On Mon, Mar 14, 2005 at 08:14:39PM -0500, sean finney wrote: much of what you're trying to do touches a similar vein to a project i'm currently working on[1]. while unfortunately i haven't built in As far as I can see your are mostly targetting packages /using/ a database? Good work so far looking at your text. The few database using packages I tried to install did not work as good as I'd have expected... any support for ldap (only mysql/pgsql), the topics, concepts, and practices are directly relevant to your situation and i'd recommend reading through it. i'd also welcome any comments you have. I'll try to find some time. On Tue, Mar 15, 2005 at 01:36:17AM +0100, Torsten Landschoff wrote: a) the preinst checks if the database format has changed between the old version and the version that we are upgrading to is this an underlying database format change, or simply stricter schema checks? The first. Basically upstream changes the database format quite often. I am even not entirely sure if the database format stays compatible in the 2.1 or 2.2 line but I'd expect it to. The 2.2.23 Debian packages uses libdb4.3 instead of libdb4.2 as used in 2.1.x so the reload has to be done in any case. if it is a change to the db format, will the new server work with the old format (even if less optimally)? if so it might make a better quality package to leave the data in the old format and provide instructions to the admin (who will know more than you about the directory server) for how to get the new optimised format. No way. b) if it has each LDAP directory is dumped to suffix.ldif using slapcat might want pipe that through gzip/bzip2 :) Hmm, good question. On a slow system this will hit really hard for big databases. On the other hand who will run a big LDAP server on a slow machine... c) the postinst checks if an ldif file is available from the old version if this is an upgrade that will always need to happen in between 2.0/2.1 and 2.2, you should rely on the version numbers provided by dpkg instead. Instead of what? I am using the version numbers from dpkg currently. set the preinst to fail if it can't successfully dump the file, which will keep the admin in as sane of a state as possible (with a working old install) That's what I am doing in any case. d) if it is, the fix_ldif script is run to adapt the contents of the directory to the more strict checking of the new OpenLDAP server does this mean you will have to drop the contents of the ldap server before re-adding them with the correct format/syntax? Installing a new version will automatically drop the old contents. Either by the script moving it away or by the server messing with the database. e) next old data in the directory of the database is moved away so the new DB can be created that's really scary sounding. remember that some people have some Important Data in these servers. at the *very* least you'll want to give them a big scary debconf warning and the ability to quit the install. That kind of contradicts seamless upgrades. And at install time (in postinst) it is already too late in the game. They'd need to reinstall the old package anyway. f) the corrected ldif file is piped into the new directory using slapadd instead of dumping to an ldif, moving the database out of the way, and reading back in from a corrected ldif, could you do the following? - dump the data in ldif format through a pipe - pipe it through your syntax/schema checker, outputting all the violations, perhaps even as ldap modify commands Way to complicated. In fact I don't even know the exact list of incompatibilities. - if there are no violations, continue with the upgrade The old Debian configuration created a directory that does not pass the schemacheck of the new packages so it is almost guaranteed there are violations. This sounds simple. There are a lot of problems so: no it doesn't :) It is mostly working already at least for simple setups. And I did not get that many reports about upgrade problems. ad b) where is that .ldif file to be saved? For small directories not an issue (take /var/backups or something). For big directories it should be on a different disk than /var/lib/ldap with enough space to get sensible performance. somewhere under /var/cache would be appropriate, though you might want to give the admin the option via debconf to put it somewhere else. /var/cache? I'd rather not put it there. Quoting the FHS: Files located under /var/cache may be expired in an application specific manner, by the system administrator, or both. The application should always be able to recover from manual deletion of these files (generally because of a disk space shortage). No other requirements are made on the data format of the cache directories. ad c) what happens if the upgrade fails for incompatibilities in slapadd? will the next dpkg
Re: [RFC] OpenLDAP automatic upgrade
Torsten Landschoff [EMAIL PROTECTED] writes: Hi there, Hi Torsten, First to introduce myself: I currently maintain Stanford University's directory service, which is based on OpenLDAP. I also am a member of the OpenLDAP core team, and I hold down another job with Symas Corporation doing a variety of tasks, but most relevantly benchmarking their OpenLDAP based product and teaching classes on running and configuring OpenLDAP based directory services. I have some comments on your upgrade plans listed here... ;) ad d) fix_ldif is a script that tries to fix some errors in the LDAP data that are not noticed by OpenLDAP 2.0 but get detected by newer OpenLDAP breaking the slapadd. Problem is: It reads the LDIF into memory. Try that with 1GB of data... Going from OpenLDAP 2.0 to 2.1 was non-trivial. Going from 2.0 to 2.2 is very complex. Going from 2.1 to 2.2 is also non-trivial. Some issues to consider for the 2.1 to 2.2 process: 1) OpenLDAP 2.1 generally used BDB 4.1. OpenLDAP 2.2 should only be used with BDB 4.2.52+patches when using bdb or hdb as the backend (Ignore the documentation with OpenLDAP 2.2.23 stating it requires BDB 4.3, that statement is incorrect). 2) Several changes were made to how ACL's were handled between OpenLDAP 2.1 and OpenLDAP 2.2. You cannot simply upgrade the database and expect everything to work. The ACL's must also be reviewed and correctly updated. 3) OpenLDAP bdb/hdb based databases are highly dependent on a well configured DB_CONFIG file for the particular database in question. Simply doing a slapcat/slapadd is really not sufficient for ensuring that the resulting directory server will run well. A poorly configured DB_CONFIG file can lead to the slapadd process taking hours or days. In addition, there are two flags that should be set in the DB_CONFIG file before the slapadd takes place, that should then be commented out once the add is completed. 4) If someone is using ldbm as their backend database, I'd throw a warning noting that bdb/hdb are the preferred backends of OpenLDAP. ldbm is rife with issues such as not catching data inconsistencies, leading to poor directory performance and potential data loss. 5) I don't know what version of OpenLDAP 2.2 that debian is considering for inclusion, but I wouldn't go with anything less than OpenLDAP 2.2.23 based on my experiences. Essentially, the process of upgrading a directory service to OpenLDAP 2.2 is generally something that should be done by the person who runs the service. There are a host of issues that must be addressed to properly upgrade. --Quanah -- Quanah Gibson-Mount Principal Software Developer ITSS/Shared Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
hey, On Tue, Mar 15, 2005 at 10:02:23PM +0100, Torsten Landschoff wrote: As far as I can see your are mostly targetting packages /using/ a database? Good work so far looking at your text. The few database using packages I tried to install did not work as good as I'd have expected... this is true, though more precisely it's packages needing to manage databases, which i think applies to this case. The first. Basically upstream changes the database format quite often. I am even not entirely sure if the database format stays compatible in the 2.1 or 2.2 line but I'd expect it to. The 2.2.23 Debian packages uses libdb4.3 instead of libdb4.2 as used in 2.1.x so the reload has to be done in any case. that sucks. if it is a change to the db format, will the new server work with the old format (even if less optimally)? if so it might make a better No way. double sucks. might want pipe that through gzip/bzip2 :) Hmm, good question. On a slow system this will hit really hard for big databases. On the other hand who will run a big LDAP server on a slow machine... yeah, i'm wondering whether or not that actually makes sense to do, now that i'm thinking about it. you make a valid point though... i would hope there aren't any directory services running on 386's, heh. c) the postinst checks if an ldif file is available from the old version if this is an upgrade that will always need to happen in between 2.0/2.1 and 2.2, you should rely on the version numbers provided by dpkg instead. Instead of what? I am using the version numbers from dpkg currently. i think i misread your statement that if postinst finds a dump file to act on it, as opposed to if the version changes suggest it should check for a dump file. that's really scary sounding. remember that some people have some Important Data in these servers. at the *very* least you'll want to give them a big scary debconf warning and the ability to quit the install. That kind of contradicts seamless upgrades. And at install time (in postinst) it is already too late in the game. They'd need to reinstall the old package anyway. if you warn them and give the ability to opt-out in the config, you'll get all the folks who use apt (which preconfigures the packages before unpacking them). if your maintscripts automatically stop slapd during the upgrade, even failing based on their response after unpacking would be okay, as they could back up to an old version before slapd started and mangled their data. reading back in from a corrected ldif, could you do the following? - dump the data in ldif format through a pipe - pipe it through your syntax/schema checker, outputting all the violations, perhaps even as ldap modify commands Way to complicated. In fact I don't even know the exact list of incompatibilities. okay, just a thought. - if there are no violations, continue with the upgrade The old Debian configuration created a directory that does not pass the schemacheck of the new packages so it is almost guaranteed there are violations. d'oh. This sounds simple. There are a lot of problems so: no it doesn't :) It is mostly working already at least for simple setups. And I did not get that many reports about upgrade problems. that's definitely reassuring. somewhere under /var/cache would be appropriate, though you might want to give the admin the option via debconf to put it somewhere else. /var/cache? I'd rather not put it there. Quoting the FHS: i think /var/cache is the a sensible default location. the only other location that fits within the fhs is /var/tmp or /tmp, which has even more liberal (the admin should be able to delete anything) requirements. plus, the data can arguably be reconstituted by dumping the ldap database again, assuming nothing goes wrong :) if you really have to move the databases, i'd recommend against hard coding where you put it. give the admin the option of where to put it. he/she will know much more about where there's free space to put it. Yes, another debconf question :(( you could do something like - use low priority to prompt for the location (so most folks won't see it) - dump the database - if failure dumping, display a debconf note saying why the dump failed (full disk), and that setting debconf priority to low and reinstalling the package will give the option of where to dump. sean -- signature.asc Description: Digital signature
Re: [RFC] OpenLDAP automatic upgrade
sean finney [EMAIL PROTECTED] writes: The first. Basically upstream changes the database format quite often. I am even not entirely sure if the database format stays compatible in the 2.1 or 2.2 line but I'd expect it to. The 2.2.23 Debian packages uses libdb4.3 instead of libdb4.2 as used in 2.1.x so the reload has to be done in any case. Just to be very clear on this again. You do *not* want to run OpenLDAP against BDB 4.3. Releasing Debian with its OpenLDAP compiled against BDB 4.3 would be a serious mistake. --Quanah -- Quanah Gibson-Mount Principal Software Developer ITSS/Shared Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [RFC] OpenLDAP automatic upgrade
hi torsten, much of what you're trying to do touches a similar vein to a project i'm currently working on[1]. while unfortunately i haven't built in any support for ldap (only mysql/pgsql), the topics, concepts, and practices are directly relevant to your situation and i'd recommend reading through it. i'd also welcome any comments you have. On Tue, Mar 15, 2005 at 01:36:17AM +0100, Torsten Landschoff wrote: a) the preinst checks if the database format has changed between the old version and the version that we are upgrading to is this an underlying database format change, or simply stricter schema checks? if it is a change to the db format, will the new server work with the old format (even if less optimally)? if so it might make a better quality package to leave the data in the old format and provide instructions to the admin (who will know more than you about the directory server) for how to get the new optimised format. b) if it has each LDAP directory is dumped to suffix.ldif using slapcat might want pipe that through gzip/bzip2 :) c) the postinst checks if an ldif file is available from the old version if this is an upgrade that will always need to happen in between 2.0/2.1 and 2.2, you should rely on the version numbers provided by dpkg instead. set the preinst to fail if it can't successfully dump the file, which will keep the admin in as sane of a state as possible (with a working old install) d) if it is, the fix_ldif script is run to adapt the contents of the directory to the more strict checking of the new OpenLDAP server does this mean you will have to drop the contents of the ldap server before re-adding them with the correct format/syntax? e) next old data in the directory of the database is moved away so the new DB can be created that's really scary sounding. remember that some people have some Important Data in these servers. at the *very* least you'll want to give them a big scary debconf warning and the ability to quit the install. f) the corrected ldif file is piped into the new directory using slapadd instead of dumping to an ldif, moving the database out of the way, and reading back in from a corrected ldif, could you do the following? - dump the data in ldif format through a pipe - pipe it through your syntax/schema checker, outputting all the violations, perhaps even as ldap modify commands - if there are no violations, continue with the upgrade - otherwise leave this in a file somewhere, and try to perform the commands - if this fails, tell the admin where the file is and let them perform the modifications before they upgrade (assuming this will not break a 2.0 server), and fail the install gracefully. This sounds simple. There are a lot of problems so: no it doesn't :) ad b) where is that .ldif file to be saved? For small directories not an issue (take /var/backups or something). For big directories it should be on a different disk than /var/lib/ldap with enough space to get sensible performance. somewhere under /var/cache would be appropriate, though you might want to give the admin the option via debconf to put it somewhere else. ad c) what happens if the upgrade fails for incompatibilities in slapadd? will the next dpkg --configure slapd give the right value for previous version to the postinst? like i said earlier, you'll want the upgrade to fail as early as possible, ideally in the preinst before you've unpacked the latest version. this way every call to dpkg --configure will provide the same values for the current and old version, and failure won't leave the admin with a totally b0rken system. ad e) where to move the directory? Should be on the same disk so that the mv command is most effective. if you really have to move the databases, i'd recommend against hard coding where you put it. give the admin the option of where to put it. he/she will know much more about where there's free space to put it. hth, sean [1] http://people.debian.org/~seanius/policy/dbapp-policy.html -- signature.asc Description: Digital signature
Re: [RFC] OpenLDAP automatic upgrade
ad b) where is that .ldif file to be saved? For small directories not an issue (take /var/backups or something). For big directories it should be on a different disk than /var/lib/ldap with enough space to get sensible performance. I think that people who are running large directories should not do such a serious upgrade silently. Preinst should issue a big fat warning that incompatable upgrade is going to happen, and probably stop slapd immidiatly. Then user should be asked if he wants to attempt automatic directory and configuration upgrade. If user agrees, package may assume that there is enough disk space. If there is not, package may detect this, inform user and and return to the question about automatic upgrade, letting user either to free some disk space or to enter path to use for ldif files or to abort upgrade or to do things manually If user does not agree, upgrade should not touch directory or configuration, and should not start slapd after upgrade. If possible, old slapcat (and libldap?) should be kept in temporary location, to make it possible to do manual directory upgrade after package upgrade. If that is not possible, package should issue a big fat warning that if .ldif files are not yet created, it will be not possible after upgrade (and directory content may be lost), and let user to stop upgrade. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
