Re: Proposed new requirements for emacsen add-on packages
On Wed, Jan 22, 2014 at 12:39:27PM -0600, Rob Browning wrote: Agustin Martin agustin6mar...@gmail.com writes: Did not test in depth, but I think the add-on state files state files could be recreated from /usr/lib/emacsen-common/packages/install in case emacsen-common is installed for the first time (i.e., not upgraded). Since emacs flavours depend on emacsen-common they should byte compile things when configured. I may be misunderstanding, but if not, I think the problem there is simultaneous installs (which are our primary complication in general), i.e. most of the complexity we have comes from the fact that we're operating outside dpkg's dependency system (to avoid add-on deps), and so we have no ordering guarantees. That's why I added the state/package/installed infrastructure, but I'm beginning to think that this approach may just be insufficient. The original idea was that the state files would allow add-ons to signal that they're ready to go with respect to emacsen-common install/remove calls. For example, consider the case where no emacsen-related packages are installed and someone runs this: apt-get install emacs24 add-on-1 Because there are no relevant dpkg deps, the packages' maintainer scripts can fire in any relative order, and if emacs24's postinst fires before add-on-1's, it can know to skip installing add-on-1 because package/installed/add-on-1 doesn't exist. But emacs24 *will* create flavor/installed/emacs24, so that when add-on-1's postinst fires, its install script will be run for emacs24. Of course, something reasonable should also happen if the order is reversed. However, to demonstrate why I'm beginning to think the current approach may be unworkable, consider the case where no emacsen-related packages are installed and someone runs this: apt-get install emacs24 add-on-depending-on-add-on-1 add-on-1 Hi, I may be wrong, but if add-on-depending-on-add-on-1 depends on add-on-1 dpkg should care of configuring add-on-1 first. However, there is no warranty about unpack ordering. I begin to suspect that up to now, we may have just been fairly lucky, and that the current approach is may just be broken for any number of cases like this. And unfortunately, adding the emacsen-common dependency won't help. Another reason why I keep wondering about the possibility of using triggers is that even if we addressed some of the above issues, the current approach will still probably result in redundant installs from time to time. Even if you want to be conservative and expect to reinstall an add-on whenever it's upgraded, or any flavor is upgraded, consider the previous example, but assume that all of the packages are already installed (just being upgraded): apt-get install emacs24 add-on-depending-on-add-on-1 add-on-1 Currently, the emacs24 postinst will install both of the add-ons, and then each add-on will install itself, and add-on-depending-on-add-on-1 will also reinstall add-on-1 again. With a triggers-based approach you'd hope that we could coordinate better, and only run each install once. Some add-ons do a check to avoid that, check that .elc for last byte-compiled file is present. If so, it is already done, otherwise byte-compile. But is way better if is emacsen-common who cares about that. So, generally speaking, unless we come up with alternatives, I'm beginning to wonder if triggers may be our best option, assuming we can shoehorn all add-ons into accommodating the attendant restrictions (whatever they end up being). I think you are right, they may be the best option, but some ellaboration is needed to properly evaluate that possibility, before going further into it. In a quick proposal, I'd think about emacs-install and emacs-package-install just touching action flags like $flavour___$package when run under dpkg control and not triggered and also enable the common emacsen-common trigger. If run from the command line or from dpkg-reconfigure (that is, not under dpkg control) they should do their usual current work and remove/update flags as needed in success. When the trigger is run, some script runs emacs-package-install as --triggered, old .elc files removed, byte-compilation done after flags and flag removed on success. I wrote some code to deal with aspell and ispell dictionaries automatic hash rebuild from dictionaries-common triggers that may be useful here (perl is also used). I'm not sure if I already mentioned it here, but I'm not opposed to triggers. Though if we made a change that substantial, we probably *would* want to move to emacsen-common 3.*. In package version yes, but there is no need to change compatibility level unless maintainers need to do something in their packages to accomodate for the new system. The problem I have with triggers to date is that I haven't yet been able to convince myself that they're flexible enough to handle all of our (potential)
Re: Proposed new requirements for emacsen add-on packages
Agustin Martin agmar...@debian.org writes: I may be wrong, but if add-on-depending-on-add-on-1 depends on add-on-1 dpkg should care of configuring add-on-1 first. However, there is no warranty about unpack ordering. No, of course -- I think you're right, and I believe that was part of my reasoning when working on the 2.* system originally. I think I just got caught up second-guessing the earlier work. So I suppose the short-term question is, assuming we do add the emacsen-common dependency, are there still any significantly broken cases? i.e. do we have to have a major overhaul in the short-term, or would the dependency be sufficient? I'll give it some further thought, but would definitely appreciate some additional evaluations/opinions. And I'd like to ignore the redundant install issue for the moment, since that's not a critical problem, just a potential waste of some time -- though certainly worth fixing if/when we can. Some add-ons do a check to avoid that, check that .elc for last byte-compiled file is present. If so, it is already done, otherwise byte-compile. But is way better if is emacsen-common who cares about that. Right, I'd originally expected that some packages might choose to be smarter, but I also wondered if there really was any safe definition of smarter. For example, to be conservative, should an add-on always rebuild? i.e. if the install script is being invoked because a new version of emacs24 is being installed, there's no way for the add-on to know whether or not the new emacs24 includes bug-fixes for byte-compilation. I think you are right, they may be the best option, but some ellaboration is needed to properly evaluate that possibility, before going further into it. Absolutely -- and first, I'd need to understand triggers a lot better. In a quick proposal, I'd think about emacs-install and emacs-package-install just touching action flags like I'll have to reread this bit after I've studied triggers more carefully. Note however that if depends is the way to go, you will have to handle this depends with a lot of maintainers with very different responsiveness maintaining add-ons with very different complexity and userbase. Agreed, though perhaps this would be a simple enough change (just the dep -- though the other cleanups the dep allows would be nice) that NMUs could be used in cases where the maintainer isn't able to get to it soonish. Currently I'm inclined to think that if we can convince ourselves that adding the dependency will more or less fix things for new-style packages, then that may be the right thing to do for now, since the alternative appears to be a much bigger overhaul -- an overhaul that I suspect in the end may well require even more substantial changes to all add-on packages. Many thanks for helping me think this through. -- Rob Browning rlb @defaultvalue.org and @debian.org GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4 ___ pkg-multimedia-maintainers mailing list pkg-multimedia-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers
Re: Proposed new requirements for emacsen add-on packages
Hi Rob It looks like your proposal is the best way forward, so I am in favour of adding this dependency in all the Emacs-dependent packages. BR -- Danai On 20 January 2014 14:20, Stephane Crivisier steph...@crivisier.fr wrote: -- Stephane Crivisier On Mon, Jan 20, 2014 at 2:43 AM, Rob Browning r...@defaultvalue.orgwrote: Recently I've been fixing some non-trivial problems I introduced in emacsen-common 2.0.0 -- and to finish fixing them it looks like it may be best to change (and augment) some of the add-on package requirements. Originally, I'd really tried to make it so that as of emacsen-common 2.*, add-on packages didn't have to depend on *anything*, but that's proving difficult to unworkable, so I'm leaning toward adding a requirement that add-on packages depend on emacsen-common = 2.0.8. If it helps, emacsen-common is only about 140k installed. Here's what I have so far from the hypothetical 2.0.8 changelog: Require add-on packages to depend on emacsen-common = 2.0.8. This should be simpler and safer, and emacsen-common is only ~140k, which shouldn't be too big a burden. One specific problem this solves is the handling of /var/lib/emacsen-common -- in particular /var/lib/emacsen-common/state/package/installed/* if/when emacsen-common is purged. Without the dependency, emacsen-common can't remove the tree without clobbering the state for every add-on, but if emacsen-common can't remove it, who can? It seems better to add this requirement for now (which should also simplify the emacsen infrastructure in general), than to have every add-on try to decide when it's safe to remove /var/lib/emacsen-common/state/package (i.e. when they're the last add-on being removed from the system). This release changes the following requirements for add-on packages (see debian-emacs-policy for the details): - They must now depend on emacsen-common = 2.0.8. - They don't need to conflict with emacsen-common anymore. - They don't need to guard their calls to emacs-install-package. - They don't need to guard their calls to emacs-remove-package. - They should no longer manage their package/installed/ file directly. In addition emacsen flavor packages should now depend on emacsen-common = 2.0.8. Thoughts? Strong objections? (And for whatever it's worth, I've been posting some relevant bits to debian-emac...@lists.debian.org lately, but I imagined that many/most of you aren't subscribed.) Thanks -- Rob Browning rlb @defaultvalue.org and @debian.org GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4 ___ pkg-multimedia-maintainers mailing list pkg-multimedia-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers
Re: Proposed new requirements for emacsen add-on packages
2014/1/20 Rob Browning r...@defaultvalue.org: Recently I've been fixing some non-trivial problems I introduced in emacsen-common 2.0.0 -- and to finish fixing them it looks like it may be best to change (and augment) some of the add-on package requirements. Originally, I'd really tried to make it so that as of emacsen-common 2.*, add-on packages didn't have to depend on *anything*, but that's proving difficult to unworkable, so I'm leaning toward adding a requirement that add-on packages depend on emacsen-common = 2.0.8. If it helps, emacsen-common is only about 140k installed. Thoughts? Strong objections? Already replied to debian-emacsen, but since many emacsen add-on maintainers may not be subscribed I am also replying here Did not test in depth, but I think the add-on state files state files could be recreated from /usr/lib/emacsen-common/packages/install in case emacsen-common is installed for the first time (i.e., not upgraded). Since emacs flavours depend on emacsen-common they should byte compile things when configured. If this works, I'd prefer this to be sorted out from emacsen-common package instead of making all these many add-on packages change. Note that this may not fix something like # dpkg --purge --force-all emacsen-common # apt-get install emacsen-common if the emacs flavour is already installed. But adding the dependency seems to also not fix it. If something like this does not work adding the dependency is the lesser evil as long as emacsen-common NEVER tries to pull emacsen nor anything that should not be present in a very minimal system. But I still prefer a fix into emacsen-common itself if possible. Other interesting things were proposed in this thread like having support for things like elfiles=foo.el bar.el autoload.el . /usr/lib/emacsen-common/emacs-install-standard.sh but while highly desirable this is something different from the original problem this thread is about. (And for whatever it's worth, I've been posting some relevant bits to debian-emac...@lists.debian.org lately, but I imagined that many/most of you aren't subscribed.) Maintainers of add-on packages, please really consider subscribing to this list if not already subscribed. It is a low traffic mailing list and is where this discussion should really take place. Regards, ___ pkg-multimedia-maintainers mailing list pkg-multimedia-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers
Re: Proposed new requirements for emacsen add-on packages
Agustin Martin agustin6mar...@gmail.com writes: Did not test in depth, but I think the add-on state files state files could be recreated from /usr/lib/emacsen-common/packages/install in case emacsen-common is installed for the first time (i.e., not upgraded). Since emacs flavours depend on emacsen-common they should byte compile things when configured. I may be misunderstanding, but if not, I think the problem there is simultaneous installs (which are our primary complication in general), i.e. most of the complexity we have comes from the fact that we're operating outside dpkg's dependency system (to avoid add-on deps), and so we have no ordering guarantees. That's why I added the state/package/installed infrastructure, but I'm beginning to think that this approach may just be insufficient. The original idea was that the state files would allow add-ons to signal that they're ready to go with respect to emacsen-common install/remove calls. For example, consider the case where no emacsen-related packages are installed and someone runs this: apt-get install emacs24 add-on-1 Because there are no relevant dpkg deps, the packages' maintainer scripts can fire in any relative order, and if emacs24's postinst fires before add-on-1's, it can know to skip installing add-on-1 because package/installed/add-on-1 doesn't exist. But emacs24 *will* create flavor/installed/emacs24, so that when add-on-1's postinst fires, its install script will be run for emacs24. Of course, something reasonable should also happen if the order is reversed. However, to demonstrate why I'm beginning to think the current approach may be unworkable, consider the case where no emacsen-related packages are installed and someone runs this: apt-get install emacs24 add-on-depending-on-add-on-1 add-on-1 What should happen if the postinsts fire in the order listed? When add-on-depending-on-add-on-1's postinst fires, add-on-1 isn't installed, and so add-on-depending-on-add-on-1 should be skipped (though that may actually be broken right now), and then when add-on-1's postinst fires, it's not (currently) going to do anything about the other add-on because add-on-depending-on-add-on-1 isn't a dependency for add-on-1. I begin to suspect that up to now, we may have just been fairly lucky, and that the current approach is may just be broken for any number of cases like this. And unfortunately, adding the emacsen-common dependency won't help. Another reason why I keep wondering about the possibility of using triggers is that even if we addressed some of the above issues, the current approach will still probably result in redundant installs from time to time. Even if you want to be conservative and expect to reinstall an add-on whenever it's upgraded, or any flavor is upgraded, consider the previous example, but assume that all of the packages are already installed (just being upgraded): apt-get install emacs24 add-on-depending-on-add-on-1 add-on-1 Currently, the emacs24 postinst will install both of the add-ons, and then each add-on will install itself, and add-on-depending-on-add-on-1 will also reinstall add-on-1 again. With a triggers-based approach you'd hope that we could coordinate better, and only run each install once. So, generally speaking, unless we come up with alternatives, I'm beginning to wonder if triggers may be our best option, assuming we can shoehorn all add-ons into accommodating the attendant restrictions (whatever they end up being). If something like this does not work adding the dependency is the lesser evil as long as emacsen-common NEVER tries to pull emacsen nor anything that should not be present in a very minimal system. But I still prefer a fix into emacsen-common itself if possible. Right -- if we go this way, I'd plan to keep emacsen-common very small, and as independent as possible. Other interesting things were proposed in this thread like having support for things like elfiles=foo.el bar.el autoload.el . /usr/lib/emacsen-common/emacs-install-standard.sh but while highly desirable this is something different from the original problem this thread is about. Agreed, and there are a number of ways we might manage something like that, which might or might not involve the code going directly into emacsen-common -- i.e. we might want a separate binary package that the relevant add-ons could depend on. Thanks -- Rob Browning rlb @defaultvalue.org and @debian.org GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4 ___ pkg-multimedia-maintainers mailing list pkg-multimedia-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers
Re: Proposed new requirements for emacsen add-on packages
[Apologies for any duplication: resending because the previous cc list had been truncated.] Modestas Vainius mo...@debian.org writes: I suggest that you bump policy version to 3 since current policy is probably going to be nothing like v2. We could, though I hadn't been planning on it since these changes only fix the current policy to work as I'd originally intended (i.e. correctly). I think the 2.* policy before these changes is likely just broken. Along those same lines, I wasn't planning to change the compat version requirement (i.e. from 0 to 1) because at the moment, I haven't thought of any way that would help us -- i.e. I can't think of any decisions that emacsen-common could make that would help, based on that number. However, it's really unfortunate that emacsen-common dependency is needed again. I know nothing about emacsen-common internals but it's kind of weird that the problem cannot be solved with dpkg triggers. I'm not sure if I already mentioned it here, but I'm not opposed to triggers. Though if we made a change that substantial, we probably *would* want to move to emacsen-common 3.*. The problem I have with triggers to date is that I haven't yet been able to convince myself that they're flexible enough to handle all of our (potential) cases. For example, with triggers, can we arrange it so that every add-on (or flavor) gets a chance to remove itself while still fully configured? And if not, do we think that we can change policy in a way that will still accommodate the needs of all add-ons (i.e. would some kind of generic removal be sufficient in all cases)? Another question -- assuming triggers run later, can all all-on packages be adjusted to behave sanely enough in the window between when they're postinst fires, and when the triggers eventually run? In my cmake case, the package just ships syntax highlighting for emacs and some users have complained about cmake pulling in anything emacs related just because of this. Understood -- one of the main points of 2.* was to remove the dependencies, but it looks like that just may not be workable with the current strategy (though I'd be happy to figure out I'm wrong). That said, an emacsen-common dependency should still be much better than what we required before, since emacsen-common is vastly smaller than any of the flavors. Ideally, the dependency requirement, assuming we decide that it's necessary, will turn out to be a temporary addition -- until we figure out how to remove it again (perhaps with triggers, or some other approach). But in the very short term, I'm just hoping to unbreak 2.*. Thanks -- Rob Browning rlb @defaultvalue.org and @debian.org GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4 ___ pkg-multimedia-maintainers mailing list pkg-multimedia-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers
Re: Proposed new requirements for emacsen add-on packages
Rob Browning r...@defaultvalue.org writes: However, to demonstrate why I'm beginning to think the current approach may be unworkable, consider the case where no emacsen-related packages are installed and someone runs this: apt-get install emacs24 add-on-depending-on-add-on-1 add-on-1 What should happen if the postinsts fire in the order listed? When add-on-depending-on-add-on-1's postinst fires, add-on-1 isn't installed, and so add-on-depending-on-add-on-1 should be skipped (though that may actually be broken right now), and then when add-on-1's postinst fires, it's not (currently) going to do anything about the other add-on because add-on-depending-on-add-on-1 isn't a dependency for add-on-1. Oh, and of course, one way you might try to fix this is arrange to have every add-on install every add-on that depends on it (and is ready). So then every add-on would be trying to install both its upstream and downstream dependencies. But even if we could do that sanely, we'd end up running even more redundant installs. -- Rob Browning rlb @defaultvalue.org and @debian.org GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4 ___ pkg-multimedia-maintainers mailing list pkg-multimedia-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers
Proposed new requirements for emacsen add-on packages
Recently I've been fixing some non-trivial problems I introduced in emacsen-common 2.0.0 -- and to finish fixing them it looks like it may be best to change (and augment) some of the add-on package requirements. Originally, I'd really tried to make it so that as of emacsen-common 2.*, add-on packages didn't have to depend on *anything*, but that's proving difficult to unworkable, so I'm leaning toward adding a requirement that add-on packages depend on emacsen-common = 2.0.8. If it helps, emacsen-common is only about 140k installed. Here's what I have so far from the hypothetical 2.0.8 changelog: Require add-on packages to depend on emacsen-common = 2.0.8. This should be simpler and safer, and emacsen-common is only ~140k, which shouldn't be too big a burden. One specific problem this solves is the handling of /var/lib/emacsen-common -- in particular /var/lib/emacsen-common/state/package/installed/* if/when emacsen-common is purged. Without the dependency, emacsen-common can't remove the tree without clobbering the state for every add-on, but if emacsen-common can't remove it, who can? It seems better to add this requirement for now (which should also simplify the emacsen infrastructure in general), than to have every add-on try to decide when it's safe to remove /var/lib/emacsen-common/state/package (i.e. when they're the last add-on being removed from the system). This release changes the following requirements for add-on packages (see debian-emacs-policy for the details): - They must now depend on emacsen-common = 2.0.8. - They don't need to conflict with emacsen-common anymore. - They don't need to guard their calls to emacs-install-package. - They don't need to guard their calls to emacs-remove-package. - They should no longer manage their package/installed/ file directly. In addition emacsen flavor packages should now depend on emacsen-common = 2.0.8. Thoughts? Strong objections? (And for whatever it's worth, I've been posting some relevant bits to debian-emac...@lists.debian.org lately, but I imagined that many/most of you aren't subscribed.) Thanks -- Rob Browning rlb @defaultvalue.org and @debian.org GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4 ___ pkg-multimedia-maintainers mailing list pkg-multimedia-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers
