Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sat, 19 Apr 2003 19:33:21 -0400, Matt Zimmerman [EMAIL PROTECTED] said: As was explained in detail, in order to do anything useful with that information, it is necessary to be able to show the user the proposed changes to the configuration file. It is completely unhelpful to say: You have modified this configuration file, and it has also been updated by the package maintainer. Do you want to replace it with the version provided by the package maintainer? Without showing the user the new version. Hmm. ucf does show the user the changes, and even offers to merge maintainer changes into the current configuration file. What functionality do you think ucf is missing? manoj -- quit When the quit statement is read, the bc processor is terminated, regardless of where the quit state- ment is found. For example, if (0 == 1) quit will cause bc to terminate. (Seen in the manpage for bc. Note the if statement's logic) Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024R/C7261095 print CB D9 F4 12 68 07 E4 05 CC 2D 27 12 1D F5 E8 6E 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sun, Apr 20, 2003 at 02:45:32AM -0500, Manoj Srivastava wrote: Hmm. ucf does show the user the changes, and even offers to merge maintainer changes into the current configuration file. What functionality do you think ucf is missing? In my first message, I listed bullet points for goals, most of which ucf meets, and then outlined the problems with this model, and linked to previous threads discussing them in detail. http://lists.debian.org/debian-devel/2003/debian-devel-200304/msg01320.html -- - mdz
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sun, 20 Apr 2003 12:22:31 -0400, Matt Zimmerman [EMAIL PROTECTED] said: On Sun, Apr 20, 2003 at 02:45:32AM -0500, Manoj Srivastava wrote: Hmm. ucf does show the user the changes, and even offers to merge maintainer changes into the current configuration file. What functionality do you think ucf is missing? In my first message, I listed bullet points for goals, most of which ucf meets, and then outlined the problems with this model, and linked to previous threads discussing them in detail. http://lists.debian.org/debian-devel/2003/debian-devel-200304/msg01320.html From my reading of that message, about the only thing that is missing is using debconf to ask the questions. Have I missed anything? (I must confess I only skimmed the first few layers of the message tree you pointed to as references; from my memory of those discussions, there was little new, and the consensus seemed to have been reached for post-inst prompting). Using debconf is on the TODO list for ucf, and perhaps a rewrite of the current prototype in C for speed later down the line. manoj -- Smear the road with a runner!! Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024R/C7261095 print CB D9 F4 12 68 07 E4 05 CC 2D 27 12 1D F5 E8 6E 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sun, Apr 20, 2003 at 12:59:27PM -0500, Manoj Srivastava wrote: On Sun, 20 Apr 2003 12:22:31 -0400, Matt Zimmerman [EMAIL PROTECTED] said: In my first message, I listed bullet points for goals, most of which ucf meets, and then outlined the problems with this model, and linked to previous threads discussing them in detail. http://lists.debian.org/debian-devel/2003/debian-devel-200304/msg01320.html From my reading of that message, about the only thing that is missing is using debconf to ask the questions. Have I missed anything? (I must confess I only skimmed the first few layers of the message tree you pointed to as references; from my memory of those discussions, there was little new, and the consensus seemed to have been reached for post-inst prompting). Yes, debconf would seem to be the only item in that list that ucf does not implement, now that the three-way option is documented in 0.12 (18 Apr). Using debconf is on the TODO list for ucf, and perhaps a rewrite of the current prototype in C for speed later down the line. ucf still has the same fundamental problem with regard to preconfiguration, which was the primary issue that I raised in my original message. The consensus, as I recall, was that preconfiguration is important, and that prompting in postinst should be minimized. If such a system is to be used for a large number of packaging, this would mean moving a large number of prompts into postinst, which I think is undesirable. -- - mdz
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
ucf still has the same fundamental problem with regard to preconfiguration, which was the primary issue that I raised in my original message. The consensus, as I recall, was that preconfiguration is important, and that prompting in postinst should be minimized. I may have missed something but why can't the changed/merged configuration files be saved somewhere in preinstall phase and the current live configs be overwritten only in the postinstall phase by dpkg or postinst scripts if the rest of the installation went well? - Jarno
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sun, Apr 20, 2003 at 09:48:48PM +0300, Jarno Elonen wrote: ucf still has the same fundamental problem with regard to preconfiguration, which was the primary issue that I raised in my original message. The consensus, as I recall, was that preconfiguration is important, and that prompting in postinst should be minimized. I may have missed something but why can't the changed/merged configuration files be saved somewhere in preinstall phase and the current live configs be overwritten only in the postinstall phase by dpkg or postinst scripts if the rest of the installation went well? Again: see my first message and followups for a specific, concrete example of why this won't work. -- - mdz
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sun, 20 Apr 2003 21:48:48 +0300, Jarno Elonen [EMAIL PROTECTED] said: ucf still has the same fundamental problem with regard to preconfiguration, which was the primary issue that I raised in my original message. The consensus, as I recall, was that preconfiguration is important, and that prompting in postinst should be minimized. I may have missed something but why can't the changed/merged configuration files be saved somewhere in preinstall phase and the current live configs be overwritten only in the postinstall phase by dpkg or postinst scripts if the rest of the installation went well? Well, for configuration files that require the unpacked package to generate, you can't ask during preconfiguration. For files created using non essential packages, you need to wait until postinst, or else those utilities may not be available. Thus, in a number of cases, the ``new'' configuration file is not determinable in the preinst phase. Even for files in the package there is no way for ucf to get at them -- especially given that the files handled by ucf are not conffiles, and thus mingled in the package data. manoj -- Virtue is a relative term. Spock, Friday's Child, stardate 3499.1 Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024R/C7261095 print CB D9 F4 12 68 07 E4 05 CC 2D 27 12 1D F5 E8 6E 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
I may have missed something but why can't the changed/merged configuration files be saved somewhere in preinstall phase [...] Again: see my first message and followups for a specific, concrete example of why this won't work. Thanks, I read the thread. So the reason was that configuration file generation is mostly done in postinst scripts? I didn't quite get why it couldn't in practically all cases be done in preinst (or even a completely new installation if required), though. Could you provide some counter-example(s)? I doubt the (pre-)dependencies on tools used for generation would really be a problem - the generation scripts are usually quite simple and can usually use pretty standard tools that don't change much. - Jarno
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
I may have missed something but why can't the changed/merged configuration files be saved somewhere in preinstall phase and the [...] Well, for configuration files that require the unpacked package to generate, you can't ask during preconfiguration. For files created using non essential packages, you need to wait until postinst, or else those utilities may not be available. I have a feeling that these cases are actually quite rare. At least with a little help, the developers currently facing those few cases would most likely be able to split their packages so that proper pre-depends could be applied. Or am I provably being too optimistic? - Jarno
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sun, 20 Apr 2003 23:49:50 +0300, Jarno Elonen [EMAIL PROTECTED] said: Thanks, I read the thread. So the reason was that configuration file generation is mostly done in postinst scripts? I didn't quite get why it couldn't in practically all cases be done in preinst (or even a completely new installation if required), though. Could you provide some counter-example(s)? my package contains /usr/bin/floobargle that I use to analize the system to create a configuration file. /usr/bin/floobargle is not available until unpacked. Or, I have a file I can generate, which is in /usr/lib/foo/bar.conf; and this file is large, and changes often, I do not want to embed it inline in my packages preinst file. I doubt the (pre-)dependencies on tools used for generation would really be a problem - the generation scripts are usually quite simple and can usually use pretty standard tools that don't change much. It depends on what I use to generate the file. A few xml files, a little bit of xsl transofrms, and you have a ton of prepends. manoj -- I was recently on a tour of Latin America, and the only regret I have was that I didn't study Latin harder in school so I could converse with those people. Danforth Quayle Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024R/C7261095 print CB D9 F4 12 68 07 E4 05 CC 2D 27 12 1D F5 E8 6E 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
* Matt Zimmerman There was a more recent discussion about the same idea. A summary of the goals: - Don't try to parse every program's configuration file format - Notice that a non-conffile, autogenerated configuration file has been modified by the user, and don't lose their changes - Provide 3-way merge functionality to incorporate changes without losing modifications in the common case (I hear this is coming for conffiles as well) - Use debconf for prompting - Have all packages do this using a standard toolset, so that all prompting is consistent and well-translated, and certain defaults can be set globally (see the threads below for more discussion about this) Hey, you just described how how ucf can be used. Here's a crash course on how to use it in package fnord's maintainer scripts: fnord.config: db_input fnord/something_that_has_no_good_default_answer db_go fnord.postinst: db_get fnord/something_that_has_no_good_default_answer cat _eof /usr/share/fnord/managed-conffiles/fnord.cf # configuration file for fnord. setting_with_acceptable_default1 = quux setting_with_acceptable_default2 = zoot # this variable hasn't got any good default. variable_with_no_good_default = $RET _eof db_stop ucf /usr/share/fnord/managed-conffiles/fnord.cf /etc/fnord.cf /dev/tty Lo and behold! We've just achieved your goals, using tools already in the archive! If /usr/share/fnord/managed-conffiles/fnord.cf changes, because of a dpkg-reconfigure where the user gives another answer, or a upgrade where the template has been changed by the maintainer (or a combination), and the user has changed /etc/fnord.cf, ucf will pop up a question which closely resembles dpkg's counterpart: Configuration file `/void/home/tore/ucftest/cf' == File on system created by you or by a script. == File also in package provided by package maintainer. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions (...) Local modification will be kept, if the user wants them to be! So there's no reason anyone to supply configuration files with comments such as ## don't edit this file; it belongs to the maintainer scrips!, even if the file is dynamically created. The user doesn't need to know that whether or not not a conffile or a ucf-handled configuration file. But wait! There's more! -- if you supply the --three-way option to ucf in your postinst, the rest of ucf's question looks like this: 3 or T : show a thre way difference between current, older, and new versions of the file M : Do a 3 way merge between current, older, and new versions of the file [Very Experimental] Exactly what you wanted, yes? -- Tore Anderson
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sat, 2003-04-19 at 15:41, Tore Anderson wrote: cat _eof /usr/share/fnord/managed-conffiles/fnord.cf /var signature.asc Description: This is a digitally signed message part
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sat, Apr 19, 2003 at 09:41:58PM +0200, Tore Anderson wrote: Hey, you just described how how ucf can be used. I am aware of ucf. I described some things that ucf does, and some things that it does not. Lo and behold! We've just achieved your goals, using tools already in the archive! Not exactly. Exactly what you wanted, yes? Did you read my sample configuration scenario (xserver-xfree86), or the threads that I referenced? They explain in more detail. -- - mdz
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
* Matt Zimmerman Did you read my sample configuration scenario (xserver-xfree86), or the threads that I referenced? They explain in more detail. I did, and I can't see why ucf can't be done for this purpose, too; As I said, I am suggesting we mimick the conffile mechanism. conffiles are not parsed, but their modification is noticed. My proposed system would not prevent the user from using the menu-driven configuration; it would simple offer them a choice about whether to generate a new configuration using the dialogs, or to preserve their existing configuration. This choice would only be necessary if the file had been modified by hand. As far as I know, ucf is created exactly for this purpose; to mimic dpkg's conffile handing. I assume you want to know if the configuration file is unmodified prior to asking all the debconf questions, and making use of such a command in the ucf package would unfortunately require a pre-dependency. However, such a test would be very simple (basically just checking if the md5sum of the configuration file matches the one recorded in ucf's hashfile), so it could easily be included in the config script. After you've conclusided that you can overwrite the configuration file (possibly by asking the user if the not modified check was negative), you just write the configuration file outside of /etc and call ucf on it. Possibly with UCF_FORCE_CONFNEW if you've asked explicit permission. What am I missing? -- Tore Anderson
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sun, Apr 20, 2003 at 01:05:18AM +0200, Tore Anderson wrote: As far as I know, ucf is created exactly for this purpose; to mimic dpkg's conffile handing. I assume you want to know if the configuration file is unmodified prior to asking all the debconf questions, and making use of such a command in the ucf package would unfortunately require a pre-dependency. A pre-dependency? We're talking about .config here, not .preinst. As was explained in detail, there is no way to get access to tools in non-essential packages from .config. However, such a test would be very simple (basically just checking if the md5sum of the configuration file matches the one recorded in ucf's hashfile), so it could easily be included in the config script. As was explained in detail, in order to do anything useful with that information, it is necessary to be able to show the user the proposed changes to the configuration file. It is completely unhelpful to say: You have modified this configuration file, and it has also been updated by the package maintainer. Do you want to replace it with the version provided by the package maintainer? Without showing the user the new version. What am I missing? Generating configuration files using non-Essential tools (including tools contained in the package being configured), the preference to ask all questions at preconfiguration time, and the necessity of displaying proposed changes before asking the user to confirm them. -- - mdz
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sat, Apr 19, 2003 at 03:14:34PM -0400, Matt Zimmerman wrote: - Provide 3-way merge functionality to incorporate changes without losing modifications in the common case (I hear this is coming for conffiles as well) Great! Actually what I would like (and is similar in ways to the above) is to be able to get two diffs: 1. What did I change in my config file since the last update? 2. What did the package maintainer change since the last update? So if the next version of the package comes with, say a totally different format, currently you will get a huge diff file between your file and the upstream file. This makes it hard to realize but I only changed one setting!, and that transferring to the new configuration might actually be a very easy task, of just copying that one setting across. -- Brian May [EMAIL PROTECTED]
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
* Matt Zimmerman As was explained in detail, in order to do anything useful with that information, it is necessary to be able to show the user the proposed changes to the configuration file. It is completely unhelpful to say: You have modified this configuration file, and it has also been updated by the package maintainer. Do you want to replace it with the version provided by the package maintainer? Without showing the user the new version. Of course, this question will be shown in the postinst, if the generated file differs from the previously generated one and the user has modified the one in /etc. I was more thinking along the lines of a question such as: You are upgrading from version 1.23 of package foo. Starting from this version, the configuration file layout has changed drastically, and your old one cannot be used. May I generate a new file based on your previous answers and autodetection tools, overwrite your old configuration file, and save the old one to /etc/foorc.dpkg-old? but I realise that's probably not what you had in mind. * Tore Anderson What am I missing? * Matt Zimmerman Generating configuration files using non-Essential tools (including tools contained in the package being configured), the preference to ask all questions at preconfiguration time, and the necessity of displaying proposed changes before asking the user to confirm them. I see your problem when you insist on asking on asking all questions at the configure stage -- personally, I don't think delaying the actual generating of the configuration file (and asking the question about overwriting the old file) to the postinst stage is *that* bad. All the other questions can be asked at configure phase, though. -- Tore Anderson
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sun, Apr 20, 2003 at 02:20:00AM +0200, Tore Anderson wrote: * Matt Zimmerman As was explained in detail, in order to do anything useful with that information, it is necessary to be able to show the user the proposed changes to the configuration file. It is completely unhelpful to say: You have modified this configuration file, and it has also been updated by the package maintainer. Do you want to replace it with the version provided by the package maintainer? Without showing the user the new version. Of course, this question will be shown in the postinst, if the generated file differs from the previously generated one and the user has modified the one in /etc. Yes, and this was the main question that I brought up at the end of my original message: is it worth sacrificing preconfiguration, or is there a better way? So far, there is no clear consensus. I see your problem when you insist on asking on asking all questions at the configure stage -- personally, I don't think delaying the actual generating of the configuration file (and asking the question about overwriting the old file) to the postinst stage is *that* bad. This is the sort of input that I was soliciting. Personally, I think that preconfiguration is very important, since it _greatly_ simplifies initial installation and upgrades (where a large number of questions are asked) and allows the rest of the installation to proceed unattended in the absence of errors. -- - mdz
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
* Tore Anderson I see your problem when you insist on asking on asking all questions at the configure stage -- personally, I don't think delaying the actual generating of the configuration file (and asking the question about overwriting the old file) to the postinst stage is *that* bad. * Matt Zimmerman This is the sort of input that I was soliciting. Personally, I think that preconfiguration is very important, since it _greatly_ simplifies initial installation and upgrades (where a large number of questions are asked) and allows the rest of the installation to proceed unattended in the absence of errors. I fully agree that as many questions as possible should be asked before unpacking the package. And I also agree it would better if the replace the configuration file questions also came at that point of the upgrade, but unfortunately, that isn't the status quo. dpkg asks for permission to overwrite conffiles *after* having unpacked the package in question -- and therefore you cannot dist-upgrade 300 packages, answer all questions, then take lunch and expect it to be finished when you're back. The upgrade will most likely stop after unpacking a package with a new conffile. As you probably know, preconfiguration is handled by apt-extracttemplates, while the conffile mechanism is dpkg's turf. Therefore it would require major changes to apt/dpkg to make those conffile questions appear at the same time as all the other questions asked by apt-extracttemplates. I don't see that change happening in the near future. So, where does that leave us? dpkg's Can I overwrite this conffile? questions are asked some time between the package is unpacked and the postinst is started. ucf's equivalent questions is asked in the postinst, which of course is slightly later in the process, but I highly doubt that a regular user would notice any difference. Hence, as ucf isn't especially worse than dpkg itself when it comes to interrupting upgrades after apt-extracttemplates has preconfigured packages, I don't think ucf's prompt in the postinst is that much of a problem. YMMV. -- Tore Anderson
Re: Proposed handling of generated configuration files (Re: stop the manage with debconf madness)
On Sun, Apr 20, 2003 at 04:11:29AM +0200, Tore Anderson wrote: I fully agree that as many questions as possible should be asked before unpacking the package. And I also agree it would better if the replace the configuration file questions also came at that point of the upgrade, but unfortunately, that isn't the status quo. dpkg asks for permission to overwrite conffiles *after* having unpacked the package in question -- and therefore you cannot dist-upgrade 300 packages, answer all questions, then take lunch and expect it to be finished when you're back. The upgrade will most likely stop after unpacking a package with a new conffile. While this is true, it isn't inherent in the design. conffile prompting could presumably be moved to the preconfiguration stage with relative ease. With dynamically generated configuration files, this is impossible for all but the simple substitution case. -- - mdz
