File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
Hi,
On Fri, Jun 5, 2009 at 2:05 PM, Adam Williamson wrote:
> On Fri, 2009-06-05 at 13:50 -0400, Ray Strode wrote:
>
>> > It seems to me it'd make sense to convert all these kinds of snippets
>> > into macros. Am I right, or is there a reason against doing this?
>> It would be awesome to get rid of the boilerplate. Honestly though,
>> I'd rather the solution was "just works" than "replace giant glob of
>> muck with %{glob_of_muck}"
>
> Yes indeed, this would be best in many cases. Some can't really be done
> like that, though - like the snippets for Tcl and Python to define the
> version, directories for certain types of file and so on. They're
> just...informational. Defining them as macros seems the optimal
> solution.
Right, totally agree.
>> For instance, if a file gets dropped under /usr/share/icons/something
>> rpm should run gtk-update-icon-cache /usr/share/icons/something
>> automatically.
>>
>> the gtk2 package should be able to drop a file in /usr/lib/rpm/redhat
>> that makes that happen. likewise, desktop-file-utils should be able
>> to drop a file there to make update-desktop-database get run and so
>> on.
>>
>> I don't know how hard it would be to fix rpm to allow for that though.
>
> This is definitely possible; Mandriva (I know I talk about MDV a lot,
> I'm sorry, it's what I know! :>) does this, via an implementation called
> 'file triggers'. This is not yet upstream, though.
>
> The MDV patch that implements this is:
>
> http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/rpm/current/SOURCES/rpm-4.6.0-rc1-filetriggers.patch?view=markup
>
> MDV's wiki discussion of it is here:
>
> http://wiki.mandriva.com/en/Rpm_filetriggers
>
> It appears to be something upstream has thought about several times.
> Here's an RPM 5 community discussion of it, with Jeff Johnson's
> thoughts:
>
> http://rpm5.org/community/rpm-devel/2959.html
>
> Here's a discussion from the rpm.org Wiki:
>
> http://www.rpm.org/wiki/Problems/Scriptlets
Oh neat!
> I'm not sure what the current status is for rpm.org - whether they're
> actively working on a solution for this side of things or what.
Panu, does this patch make sense?
> Oh, and please try to consider the original proposal in replies to this
> thread, as I sense a derail coming :). file triggers is certainly an
> interesting topic, but there are some snippets which just don't fit the
> case, see above.
Right, I'veretitled the subject. I guess there are two different
halves to the spec boilerplate problem.
--Ray
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
On Fri, 5 Jun 2009, Ray Strode wrote:
Hi,
On Fri, Jun 5, 2009 at 2:05 PM, Adam Williamson wrote:
On Fri, 2009-06-05 at 13:50 -0400, Ray Strode wrote:
It seems to me it'd make sense to convert all these kinds of snippets
into macros. Am I right, or is there a reason against doing this?
It would be awesome to get rid of the boilerplate. Honestly though,
I'd rather the solution was "just works" than "replace giant glob of
muck with %{glob_of_muck}"
Yes indeed, this would be best in many cases. Some can't really be done
like that, though - like the snippets for Tcl and Python to define the
version, directories for certain types of file and so on. They're
just...informational. Defining them as macros seems the optimal
solution.
Right, totally agree.
Yes, having each and every spec carry the %{!?foo:¤%&¤%} macro goo makes
no sense at all.
For instance, if a file gets dropped under /usr/share/icons/something
rpm should run gtk-update-icon-cache /usr/share/icons/something
automatically.
the gtk2 package should be able to drop a file in /usr/lib/rpm/redhat
that makes that happen. likewise, desktop-file-utils should be able
to drop a file there to make update-desktop-database get run and so
on.
I don't know how hard it would be to fix rpm to allow for that though.
The hardest part is getting the design right the first time, there's no
changing an "api" that is exposed to packages.
This is definitely possible; Mandriva (I know I talk about MDV a lot,
I'm sorry, it's what I know! :>) does this, via an implementation called
'file triggers'. This is not yet upstream, though.
The MDV patch that implements this is:
http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/rpm/current/SOURCES/rpm-4.6.0-rc1-filetriggers.patch?view=markup
MDV's wiki discussion of it is here:
http://wiki.mandriva.com/en/Rpm_filetriggers
It appears to be something upstream has thought about several times.
Here's an RPM 5 community discussion of it, with Jeff Johnson's
thoughts:
http://rpm5.org/community/rpm-devel/2959.html
Here's a discussion from the rpm.org Wiki:
http://www.rpm.org/wiki/Problems/Scriptlets
Oh neat!
I'm not sure what the current status is for rpm.org - whether they're
actively working on a solution for this side of things or what.
Panu, does this patch make sense?
I'm quite well aware of the Mandriva filetriggers patch, but I (too) have
some reservations about the implementation. Other issues aside,
constraining file triggers to "stuff that runs after the transaction
completed" limits the possibilities a lot. Many things need unregistering
(services, gconf schemas, info files...) and that needs to happen *before*
the files got removed. That's not possible in post-transaction stage.
I've been playing with a different approach: putting an existing (since
rpm 4.4.0 IIRC) but completely unused "hook" mechanism into use, this
allows pushing pretty much the entire mechanism out of rpm proper and into
Lua, which allows for far more flexibility for distros to customize etc.
The most simple variant consists of a whopping two-line patch to rpm:
http://laiskiainen.org/tmp/rpmhook-example/
This works by comparing modification times of directories of interest -
not sufficient for a real-world implementation but still works quite well
(and with essentially zero overhead) for the "update some cache on demand"
class of cases, as false positives dont generally matter there.
Things like automatically registering and unregistering info/gconf etc
files need more information than just "directory changed". What they need
is access to package file lists and the file "fate" information (such as
"these files are about to be removed"), and the ability to run the
"triggers" at start/in middle of transaction. Mostly a matter of planting
some extra hooks into strategic spots, and extending the rpm-lua
interface.
One of the nice things about the hook-based approach is that it permits
each trigger to use whatever mechanism is best suited for the job:
processing file names/globs and remembering matches can be quite
expensive, so a trigger that doesn't need the details can just use a
simple directory modification time based approach.
Also for ultimate power the file triggers need to be in headers so that
all triggers are ready for action before the transaction starts, to avoid
unnecessary dependencies and ordering issues. And then you'll need
semantics for upgrade of a package containing file triggers - you probably
dont want the trigger from both the already installed package and the
upgraded package to run.
It's a fair-sized puzzle to get all the bits sufficiently right the first
time, or alternatively expose things little by little in a manner that
doesn't paint us in a corner.
- Panu -
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
On Sat, 6 Jun 2009, Panu Matilainen wrote: Also for ultimate power the file triggers need to be in headers so that all triggers are ready for action before the transaction starts, to avoid unnecessary dependencies and ordering issues. And then you'll need semantics for upgrade of a package containing file triggers - you probably dont want the trigger from both the already installed package and the upgraded package to run. That said, there are some options that would allow introducing file triggers without actually putting them to headers. The biggest problem is initial install to an empty chroot, but it's possible to work around it to some extent. Keeping the trigger scripts initially more or less internal to rpm would give the much needed room for experimenting. Once the "api" details seem about right, the trigger scripts can be pushed to the "owning" packages where they belong. The other packages relying on file triggers wouldn't see a thing of all this, except for the initial leap of faith "rpm will arrange this somehow". If Fedora is willing to play a guinea pig here, I'm game. Of course there are transition issues... packages relying on file-trigger behavior wouldn't install correctly on older rpm's not providing the functionality. This would affect pretty much any mock builds, including Koji builders... so either the use of file-triggers should be limited to things that aren't strictly required so this doesn't really matter (most of the cache updating stuff etc falls to this category luckily) and/or the file trigger stuff is run-time conditionalized in specs, eg %post if [ -z "$RPM_HAVE_FILETRIGGERS" ]; then scrollkeeper-update gtk-update-icon-cache fi ...so initially we'd have *more* junk in specs. - Panu - -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
On Sat, 2009-06-06 at 15:47 +0300, Panu Matilainen wrote: > On Sat, 6 Jun 2009, Panu Matilainen wrote: I'm glad to see I'm not the only one who replies to himself :) > > > > Also for ultimate power the file triggers need to be in headers so that all > > triggers are ready for action before the transaction starts, to avoid > > unnecessary dependencies and ordering issues. And then you'll need > > semantics > > for upgrade of a package containing file triggers - you probably dont want > > the trigger from both the already installed package and the upgraded > > package > > to run. > If Fedora is willing to play a guinea pig here, I'm game. FWIW (as the idea grew from mine) I'm not ra-ra for triggers - I'd rather get it right slowly, as you suggest. But I guess it has to get tested _somewhere_... > Of course there are transition issues... packages relying on file-trigger > behavior wouldn't install correctly on older rpm's not providing the > functionality. This would affect pretty much any mock builds, including > Koji builders... so either the use of file-triggers should be limited to > things that aren't strictly required so this doesn't really matter (most > of the cache updating stuff etc falls to this category luckily) and/or the > file trigger stuff is run-time conditionalized in specs, eg > > %post > if [ -z "$RPM_HAVE_FILETRIGGERS" ]; then > scrollkeeper-update > gtk-update-icon-cache > fi > > ...so initially we'd have *more* junk in specs. Yeah, MDV had that problem with file triggers. For 18 months we had to keep "if you're an old version run these macros" junk in the specs. But that's never going to _not_ be the case (unless we introduce the code for triggers, then wait a year or two before using it. Or longer, in the case of packages that are dual purpose for RHEL / EPEL, I guess.) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
On Sat, 6 Jun 2009, Adam Williamson wrote: On Sat, 2009-06-06 at 15:47 +0300, Panu Matilainen wrote: On Sat, 6 Jun 2009, Panu Matilainen wrote: I'm glad to see I'm not the only one who replies to himself :) Heh :) Also for ultimate power the file triggers need to be in headers so that all triggers are ready for action before the transaction starts, to avoid unnecessary dependencies and ordering issues. And then you'll need semantics for upgrade of a package containing file triggers - you probably dont want the trigger from both the already installed package and the upgraded package to run. If Fedora is willing to play a guinea pig here, I'm game. FWIW (as the idea grew from mine) I'm not ra-ra for triggers - I'd rather get it right slowly, as you suggest. But I guess it has to get tested _somewhere_... Yup. And I think being able to test in a real distro would expose all the strange cases I haven't even thought of yet much faster than just trying to puzzle it all together on paper. Of course there are transition issues... packages relying on file-trigger behavior wouldn't install correctly on older rpm's not providing the functionality. This would affect pretty much any mock builds, including Koji builders... so either the use of file-triggers should be limited to things that aren't strictly required so this doesn't really matter (most of the cache updating stuff etc falls to this category luckily) and/or the file trigger stuff is run-time conditionalized in specs, eg %post if [ -z "$RPM_HAVE_FILETRIGGERS" ]; then scrollkeeper-update gtk-update-icon-cache fi ...so initially we'd have *more* junk in specs. Yeah, MDV had that problem with file triggers. For 18 months we had to keep "if you're an old version run these macros" junk in the specs. But that's never going to _not_ be the case (unless we introduce the code for triggers, then wait a year or two before using it. Or longer, in the case of packages that are dual purpose for RHEL / EPEL, I guess.) Btw your initial suggestion of collecting the common stuff into macros and converting packages to use them would be useful on several ways: a) Finding out the things that *are* common among lots of packages. While numerous cases are well and widely known already, I suspect there might be some that are only specific groups know about (possibly eg java related packages, I dunno). b) Making the usages of the common patterns easy to spot by grepping. c) The transition period cruft can be hidden inside the common macros without polluting every spec with it. - Panu - -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
Le samedi 06 juin 2009 à 13:01 +0300, Panu Matilainen a écrit :
> Yes, having each and every spec carry the %{!?foo:¤%&¤%} macro goo makes
> no sense at all.
That is pretty much what we did for fonts in F11. However many packagers
just ignored the change and didn't fix their packages.
> >>> For instance, if a file gets dropped under /usr/share/icons/something
> >>> rpm should run gtk-update-icon-cache /usr/share/icons/something
> >>> automatically.
> >>>
> >>> the gtk2 package should be able to drop a file in /usr/lib/rpm/redhat
> >>> that makes that happen. likewise, desktop-file-utils should be able
> >>> to drop a file there to make update-desktop-database get run and so
> >>> on.
> >>>
> >>> I don't know how hard it would be to fix rpm to allow for that though.
>
> The hardest part is getting the design right the first time, there's no
> changing an "api" that is exposed to packages.
I personally think it would be a huge mistake to have stuff happen
automatically based on filename/location heuristics. Naming collisions
happen all the time (for example GNOME recently decided that a third of
our fonts were "ODF templates" and good luck trying to find someone
ready to acknowledge it's a problem). I'd much prefer if the packager
was left in control and specified the processing himself, for example by
allowing more magic %doc-like words in %files.
In my ideal pony-land, we could have stuff like
%files somepackage
%font somefont.ttf
%icon-dir somedirectory
...
that injected the correct logic in %install/%pre/%post/%postun/%
posttrans etc
And I suppose some of those magic words would work on files already
installed, others on files in the build root (like %doc), and you'd need
them to interact (for example, consolidate three %font lines in a
package in a single actions, have the %font word be aware of the %
fontconfig word so one could correct font file processing with explicit
fontconfig rules, etc)
--
Nicolas Mailhot
signature.asc
Description: Ceci est une partie de message numériquement signée
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
Le dimanche 07 juin 2009 à 12:31 +0300, Panu Matilainen a écrit : > Btw your initial suggestion of collecting the common stuff into macros > and converting packages to use them would be useful on several ways: > a) Finding out the things that *are* common among lots of packages. While > numerous cases are well and widely known already, I suspect there might > be some that are only specific groups know about (possibly eg java > related packages, I dunno). > b) Making the usages of the common patterns easy to spot by grepping. > c) The transition period cruft can be hidden inside the common macros > without polluting every spec with it. Won't work IMHO. One characteristic of pre-macroized specs is that their authors have found lots of "interesting" ways to do about the same thing with different instructions (usually missing some problems). If you don't want this old processing to collide and interfere with your new shiny processing you need the call to the new processing to be explicit, so packagers can check for problems before enabling it. -- Nicolas Mailhot signature.asc Description: Ceci est une partie de message numériquement signée -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
On Mon, 2009-06-08 at 13:25 +0200, Nicolas Mailhot wrote: > I personally think it would be a huge mistake to have stuff happen > automatically based on filename/location heuristics. Naming collisions > happen all the time (for example GNOME recently decided that a third of > our fonts were "ODF templates" and good luck trying to find someone > ready to acknowledge it's a problem). Did you file a bug against shared-mime-info ? I don't recall seeing one go by... -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
Le lundi 08 juin 2009 à 09:34 -0400, Matthias Clasen a écrit : > On Mon, 2009-06-08 at 13:25 +0200, Nicolas Mailhot wrote: > > > I personally think it would be a huge mistake to have stuff happen > > automatically based on filename/location heuristics. Naming collisions > > happen all the time (for example GNOME recently decided that a third of > > our fonts were "ODF templates" and good luck trying to find someone > > ready to acknowledge it's a problem). > > Did you file a bug against shared-mime-info ? I don't recall seeing one > go by... I filed http://bugzilla.redhat.com/show_bug.cgi?id=491598 http://bugzilla.gnome.org/show_bug.cgi?id=576360 http://bugs.freedesktop.org/show_bug.cgi?id=20854 at which point I decided to give up on the bugzilla NOTOURBUG ping pong game. -- Nicolas Mailhot signature.asc Description: Ceci est une partie de message numériquement signée -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
On Mon, 2009-06-08 at 15:45 +0200, Nicolas Mailhot wrote: > Le lundi 08 juin 2009 à 09:34 -0400, Matthias Clasen a écrit : > > On Mon, 2009-06-08 at 13:25 +0200, Nicolas Mailhot wrote: > > > > > I personally think it would be a huge mistake to have stuff happen > > > automatically based on filename/location heuristics. Naming collisions > > > happen all the time (for example GNOME recently decided that a third of > > > our fonts were "ODF templates" and good luck trying to find someone > > > ready to acknowledge it's a problem). > > > > Did you file a bug against shared-mime-info ? I don't recall seeing one > > go by... > > I filed > > http://bugzilla.redhat.com/show_bug.cgi?id=491598 > http://bugzilla.gnome.org/show_bug.cgi?id=576360 > http://bugs.freedesktop.org/show_bug.cgi?id=20854 > > at which point I decided to give up on the bugzilla NOTOURBUG ping pong > game. > Don't give up so easily...I've reopened the right bug and recommended a fix. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
On Mon, 2009-06-08 at 13:31 +0200, Nicolas Mailhot wrote: > Le dimanche 07 juin 2009 à 12:31 +0300, Panu Matilainen a écrit : > > > Btw your initial suggestion of collecting the common stuff into macros > > and converting packages to use them would be useful on several ways: > > a) Finding out the things that *are* common among lots of packages. While > > numerous cases are well and widely known already, I suspect there might > > be some that are only specific groups know about (possibly eg java > > related packages, I dunno). > > b) Making the usages of the common patterns easy to spot by grepping. > > c) The transition period cruft can be hidden inside the common macros > > without polluting every spec with it. > > Won't work IMHO. One characteristic of pre-macroized specs is that their > authors have found lots of "interesting" ways to do about the same thing > with different instructions (usually missing some problems). If you > don't want this old processing to collide and interfere with your new > shiny processing you need the call to the new processing to be explicit, > so packagers can check for problems before enabling it. I'm not planning to re-implement anything, just change things that are currently snippets of code you have to copy and paste from the guidelines to be macros instead. All it involves is taking the snippets and putting them into a /etc/macros.* file. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
Nicolas Mailhot wrote:
Le samedi 06 juin 2009 à 13:01 +0300, Panu Matilainen a écrit :
Yes, having each and every spec carry the %{!?foo:¤%&¤%} macro goo makes
no sense at all.
That is pretty much what we did for fonts in F11. However many packagers
just ignored the change and didn't fix their packages.
...
I personally think it would be a huge mistake to have stuff happen
automatically based on filename/location heuristics.
...
I'd much prefer if the packager
was left in control and specified the processing himself, for example by
allowing more magic %doc-like words in %files.
That shows the problem very well. Full control for the packagers is
demanded but it is difficult to force the right (tm) behavior on the
packagers. The question is do we really win something by leaving the
handling of every file to the packager or do we demand to much when
requiring the knowledge how to handle every type of file.
I btw very much dislike the term "filename/location heuristics" in this
context as it gets things wrong. File triggers is not about guessing
what to do with the different files but to create, implement and then
enforce rules how files of a given type are installed. This includes
where these files need to be placed or how they are named - not exactly
a new concept for a huge number of file types.
With the growing number of packages and file types that need special
handling the burden for Fedora for handling these files is also growing.
While changing/removing 95% of the scriptlets is a huge effort it will
pay off - especially as lowers the level of entry for packaging.
In my ideal pony-land, we could have stuff like
%files somepackage
%font somefont.ttf
%icon-dir somedirectory
...
that injected the correct logic in %install/%pre/%post/%postun/%
posttrans etc
And I suppose some of those magic words would work on files already
installed, others on files in the build root (like %doc), and you'd need
them to interact (for example, consolidate three %font lines in a
package in a single actions, have the %font word be aware of the %
fontconfig word so one could correct font file processing with explicit
fontconfig rules, etc)
This approach has several shortcomings (forgetting the technical
details). It requires a lot of data be shipped with each package. Data
which can be already out of date (method of handling the files changed).
It still requires the packager to know about all file types and whether
they need special handling. And even worse it requires RPM to know about
the different file types.
The nice about file triggers is that the package that actually knows
about the file type ships the file trigger for handling it. glibc would
ship the trigger for calling ldconfig and info the trigger for info
files and so on.
Florian
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
On Sat, Jun 6, 2009 at 6:01 AM, Panu Matilainen wrote:
>
> The hardest part is getting the design right the first time, there's no
> changing an "api" that is exposed to packages.
It's definitely better to get things "right" the first time, but one
thing missing from the system now is any kind of "versioning" on the
macro API. It would seem fairly straightforward to have a global
"SpecVersion: 2" type header that pulls in an updated set of macros
from a different directory.
If such a thing were to be implemented it'd probably be good to use it
to clean out the wishlist in general, like handling %clean and even
%build automatically (e.g. if we see a configure script, just assume
to call "%{configure}", see a Makefile, just assume to call "make",
etc)
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
Le lundi 08 juin 2009 à 20:13 +0200, Florian Festi a écrit : > > This approach has several shortcomings (forgetting the technical > details). It requires a lot of data be shipped with each package. I think you misunderstood me. I'd want the definition for %font of % icon-dir to be factored-out and centralized too (and not necessarily in an rpm subpackage BTW, a %lib definition shipped with glibc would be perfectly fine with me). Also, that does not prevent standardising install locations (that's why I ask something that can hook in %install). My example, %doc, already makes file location decisions BTW. What I don't want is 1. something auto-triggered transparently (didn't we learn anything from existing package triggers?). Some of us do not have the luxury of uniform-quality input files. Therefore, I want the decision to launch the processing packager-side. I want the packager to decide and check some processing is right for his files, and not discover at QA time another packager decided his file looked like a candidate for froobing and should be froobed behind his back. The existing auto-processing (for example debuginfo generation) has been wrong so many times the web is littered with way to disable it (often with side-effects). Just because you can write good froobing rules does not mean you understand how to auto-select files to be froobed accurately. 2. something that auto-generates (sub)packages. The packager should decide how big or small his packages will be, if they deserve splitting or not, etc. Auto-package creation leads in many cases to monster packages to big to be installed in any reasonable system. -- Nicolas Mailhot signature.asc Description: Ceci est une partie de message numériquement signée -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
Colin Walters wrote:
> If such a thing were to be implemented it'd probably be good to use it
> to clean out the wishlist in general, like handling %clean and even
> %build automatically (e.g. if we see a configure script, just assume
> to call "%{configure}", see a Makefile, just assume to call "make",
> etc)
Uh, that will never work. Many configure scripts need options, even
makefiles can require options. The makefile could also be autogenerated
from an unknown build tool and accidentally shipped by upstream (in which
case you need to run that build tool first to generate a proper one).
Kevin Kofler
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
On Tue, Jun 9, 2009 at 11:22 AM, Kevin Kofler wrote:
> Colin Walters wrote:
>> If such a thing were to be implemented it'd probably be good to use it
>> to clean out the wishlist in general, like handling %clean and even
>> %build automatically (e.g. if we see a configure script, just assume
>> to call "%{configure}", see a Makefile, just assume to call "make",
>> etc)
>
> Uh, that will never work. Many configure scripts need options, even
> makefiles can require options.
There are always exceptions. That doesn't mean one can't optimize for
the common case (and not just common, but what we want to push people
to do, which is consolidate build tools).
> The makefile could also be autogenerated from an unknown build tool
Makefiles generated by e.g. automake have easily detectable patterns.
Try "head Makefile".
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
Colin Walters wrote: > On Tue, Jun 9, 2009 at 11:22 AM, Kevin Kofler > wrote: >> The makefile could also be autogenerated from an unknown build tool > > Makefiles generated by e.g. automake have easily detectable patterns. > Try "head Makefile". I wrote "UNKNOWN build tool". Of course you can detect the known ones (automake, CMake etc.), but upstream may come up with some new build tool, I've seen even custom makefile generators for single projects being used. Kevin Kofler -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
Nicolas Mailhot wrote: Le lundi 08 juin 2009 à 20:13 +0200, Florian Festi a écrit : This approach has several shortcomings (forgetting the technical details). It requires a lot of data be shipped with each package. I think you misunderstood me. I'd want the definition for %font of % icon-dir to be factored-out and centralized too (and not necessarily in an rpm subpackage BTW, a %lib definition shipped with glibc would be perfectly fine with me). Yeah, but patching the spec file parsing from out side sounds scary (Ok, may be these implementation details should be ignored at this level of disscussion). Also, that does not prevent standardising install locations (that's why I ask something that can hook in %install). My example, %doc, already makes file location decisions BTW. Sure. What I don't want is 1. something auto-triggered transparently (didn't we learn anything from existing package triggers?). I think you make the wrong comparison here (although I admit that the matching names make it tempting). Triggers fill holes in the scriptlet mechanism and though are restricted to obscure and complicated cases. The file trigger I am talking about will handle the easy, omnipresent cases where files need additional treatment. They better compare to the automatic dependency creation. As the automatic dependency creation file triggers would benefit of a good/better control by the packager. 2. something that auto-generates (sub)packages. The packager should decide how big or small his packages will be, if they deserve splitting or not, etc. Auto-package creation leads in many cases to monster packages to big to be installed in any reasonable system. This probably deserves an thread on its own... I don't think there will be any additional full automatic creation of sub packages in Fedora (hmm, but automatic -bin packages to get everything else noarch...). But semi automatic sub package creation is going to be an important part when/if we split out language sub packages. My idea is that you specify a regex for the files that go into the sub packages and a matching group that names the sub package and becomes a macro used in the package template. But before splitting out language sub packages makes sense we have to fix a lot of other places first. So much to do some so little time. Florian -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
Le Mer 10 juin 2009 10:59, Florian Festi a écrit : > > Nicolas Mailhot wrote: >> 1. something auto-triggered transparently (didn't we learn anything >> from >> existing package triggers?). > > I think you make the wrong comparison here (although I admit that the > matching names make it tempting). Triggers fill holes in the scriptlet > mechanism and though are restricted to obscure and complicated cases. The new trigger proposal has exactly the same problem as existing triggers: processing which is specified in a separate package, and happens magically if this package is available (on system or on build root), without the packager of the current package having any control on it. It will lead to exactly the same weird bugs and packager pain. Again, if you think you've factored out some cool processing function, please oh please give it a proper name/id and convince packagers to explicitely invoke this name/id in their specs do not inject code behind their backs. Wishing it all to happen transparently with no packager action is laudable, but in practice all past attempts to do so have ended up in pain for packagers and as they say the road to hell is paved with good intentions > But semi automatic sub package creation is going to be an > important part when/if we split out language sub packages. My idea is > that you specify a regex for the files that go into the sub packages > and a matching group that names the sub package and becomes a macro > used in the package template. After 8+ months factoring out font subpackage creation and being forced by rpm limitations to do some form of automatic subpackage creation I can plainly say this is a bad idea. Packagers need the subpackage declarations to hang on deps, conflicts, ancillary files like doc, etc. Packagers need the subpackage declaration to control the size of theirs packages. Even if some package source includes 100 files with the same technical characteristics that does not mean you want to create a monster 100-files subpackages (and this is not a theorical argument, see TEX for example). So, do factor out logic, do help packagers assemble subpackages by calling common routines on the files they choose, but do not try to select the files in their stead. Except for very trivial cases you're going to have fallout, limitations and other unintended side-effects all over the place. -- Nicolas Mailhot -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
On Wed, 10 Jun 2009, Nicolas Mailhot wrote:
Le Mer 10 juin 2009 10:59, Florian Festi a écrit :
Nicolas Mailhot wrote:
1. something auto-triggered transparently (didn't we learn anything
from
existing package triggers?).
I think you make the wrong comparison here (although I admit that the
matching names make it tempting). Triggers fill holes in the scriptlet
mechanism and though are restricted to obscure and complicated cases.
The new trigger proposal has exactly the same problem as existing
triggers: processing which is specified in a separate package, and
happens magically if this package is available (on system or on build
root), without the packager of the current package having any control
on it. It will lead to exactly the same weird bugs and packager pain.
Again, if you think you've factored out some cool processing function,
please oh please give it a proper name/id and convince packagers to
explicitely invoke this name/id in their specs do not inject code
behind their backs.
Wishing it all to happen transparently with no packager action is
laudable, but in practice all past attempts to do so have ended up in
pain for packagers and as they say the road to hell is paved with good
intentions
File triggers are certainly not the holy grail of packaging, they're only
applicaple to a pretty limited set of situations, from the top of my head:
1) Caches updaters which you only want to run once per transaction:
- ldconfig
- scrolkeeper-update
- gtk-update-icon-cache
- update-desktop-database
- fc-cache
2) Files in well-known locations that might be automatically registerable:
- install-info add + remove of %{_infodir}/*.info*
- init scripts (chkconfig --add/--del, service stop/condrestart)
- gconf schema install+remove
- plugin registrations
The cases in 1) are the "classic" file trigger examples, things that
aren't absolutely critical for the package itself to be runnable, and
where false positives / multiple unnecessary runs are not dangerous at
all. They're just telling some other package "please update your caches".
I dont see any point in requiring special extra magic in specs to activate
them.
The cases in 2) differ in varying degrees. Info-file
registration/unregistration seems safe enough to me: by putting an *.info*
file into %{_infodir} you are announcing it's an info file. There's not
much room for mistakes here I'd think, and it's quite close to category 1)
actually, except it needs to run at different times (to handle removal).
Services and gconf .. might not be so obvious, and whether plugin
registration/unregistration can reliably be done automatically is
case by case.
In both categories there's a big difference to the current name/provide
triggers: with file triggers you knowingly place something into some other
packages directory, so following the principles of directory ownership you
should already depend on the other package. With name/provide triggers any
completely unrelated package can do anything at all at any time. Maybe
packages should only be able to add triggers on directories they actually
own (subject to abuse too but then what isnt...).
AFAICT, you're talking what would basically be "a named trigger", to which
packages "subscribe to" if they want to. It's not at odds with file
triggers at all, and both are likely to get implemented sooner or later.
What distros choose to use for particular task is up to their packaging
committees and whatnot, rpm is to only provide a mechanism, not policy
or any magic internal triggers here.
- Panu -
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
Le Mer 10 juin 2009 13:21, Panu Matilainen a écrit :
>
> File triggers are certainly not the holy grail of packaging, they're
> only
> applicaple to a pretty limited set of situations, from the top of my
> head:
>
> 1) Caches updaters which you only want to run once per transaction:
> - ldconfig
> - scrolkeeper-update
> - gtk-update-icon-cache
> - update-desktop-database
> - fc-cache
Actually, I doubt we will ever run fc-cache once per transaction. The
consequences of bad fontconfig caches are just too high. What we've
been doing is runing fc-cache just-as-needed by making each srpm
install files in a different directory and having resulting rpms
refresh only this directory cache, instead of processing all the
system font directories each time a font package is installed.
> 2) Files in well-known locations that might be automatically
> registerable:
> - install-info add + remove of %{_infodir}/*.info*
> - init scripts (chkconfig --add/--del, service stop/condrestart)
> - gconf schema install+remove
> - plugin registrations
>
> The cases in 1) are the "classic" file trigger examples, things that
> aren't absolutely critical for the package itself to be runnable, and
> where false positives / multiple unnecessary runs are not dangerous at
> all.
Multiple runs yes, false positives do not be so sure. False positives
is the main weakness of this proposal and "good stuff will happen if
the autoselection is correct" is very different from "bad stuff won't
happen if the autoselection is false".
> They're just telling some other package "please update your
> caches".
And relying on the cache updating utilities to have ironclad false
positive protection logic. Which is not a given, since those utilities
have always been explicitely invoked with a human sanity-checking
input files before.
BTW: the system can usually manage when those caches are stale, not
when they are corrupted.
> I dont see any point in requiring special extra magic in specs to
> activate
> them.
>
> The cases in 2) differ in varying degrees. Info-file
> registration/unregistration seems safe enough to me: by putting an
> *.info*
> file into %{_infodir} you are announcing it's an info file. There's
> not
> much room for mistakes here I'd think, and it's quite close to
> category 1)
> actually, except it needs to run at different times (to handle
> removal).
This is backwards IMHO
1. it relies on all interesting files having a clear FHS location or
unambiguous file name
2. it relies on the packager guessing the right places to put his
files to trigger processing. The logic should not be "I have an info
file, let's put it here so rpm guesses it's an info file and does the
right thing". The logic should be "I tell rpm this is an info file and
rpm does the right thing, including installing it in the right place".
This is of course the POW of a packager that has to cope with
imperfect tools that try to be smarter than they can be, not the POW
of the person that writes the smart tools and is convinced he can't do
wrong.
Now, some way to register build-time trigger warnings "your package is
installing X file that seems to be Y, please consider processing it
with %_zzz y" would be nice. But that's something else entirely.
> What distros choose to use for particular task is up to their
> packaging
> committees and whatnot, rpm is to only provide a mechanism, not policy
> or any magic internal triggers here.
To put it another way: automatic file triggers are policy. Exporting
commands packagers may choose to use (or not) is a mechanism.
--
Nicolas Mailhot
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
On Wed, 10 Jun 2009, Nicolas Mailhot wrote:
Le Mer 10 juin 2009 13:21, Panu Matilainen a écrit :
File triggers are certainly not the holy grail of packaging, they're
only
applicaple to a pretty limited set of situations, from the top of my
head:
1) Caches updaters which you only want to run once per transaction:
- ldconfig
- scrolkeeper-update
- gtk-update-icon-cache
- update-desktop-database
- fc-cache
Actually, I doubt we will ever run fc-cache once per transaction. The
consequences of bad fontconfig caches are just too high. What we've
been doing is runing fc-cache just-as-needed by making each srpm
install files in a different directory and having resulting rpms
refresh only this directory cache, instead of processing all the
system font directories each time a font package is installed.
Well, I'm not intimately aware of the font handling details nor do I
want to be, I was just under the impression the font cache belongs to the
category 1). And this is why the actual script to do whatever magic it
needs to do, when it needs to, would be in a distros fontconfig package,
not rpm.
2) Files in well-known locations that might be automatically
registerable:
- install-info add + remove of %{_infodir}/*.info*
- init scripts (chkconfig --add/--del, service stop/condrestart)
- gconf schema install+remove
- plugin registrations
The cases in 1) are the "classic" file trigger examples, things that
aren't absolutely critical for the package itself to be runnable, and
where false positives / multiple unnecessary runs are not dangerous at
all.
Multiple runs yes, false positives do not be so sure. False positives
is the main weakness of this proposal and "good stuff will happen if
the autoselection is correct" is very different from "bad stuff won't
happen if the autoselection is false".
False positive in this context would mean either
a) the cache update run without needing to
b) a package put something into a wrong directory
a) is harmless as per multiple runs, b) is a grave packaging bug which
with file triggers would be caught when installing the buggy package,
instead of next cache update started by something else which then might
blow up/issue warnings long time afterwards.
They're just telling some other package "please update your
caches".
And relying on the cache updating utilities to have ironclad false
positive protection logic. Which is not a given, since those utilities
have always been explicitely invoked with a human sanity-checking
input files before.
See above, they already need an ironclad false positive protection.
BTW: the system can usually manage when those caches are stale, not
when they are corrupted.
I dont see any point in requiring special extra magic in specs to
activate
them.
The cases in 2) differ in varying degrees. Info-file
registration/unregistration seems safe enough to me: by putting an
*.info*
file into %{_infodir} you are announcing it's an info file. There's
not
much room for mistakes here I'd think, and it's quite close to
category 1)
actually, except it needs to run at different times (to handle
removal).
This is backwards IMHO
1. it relies on all interesting files having a clear FHS location or
unambiguous file name
The idea of file triggers is *based* on the "well known locations". If
something doesn't have a clear and well known location, file triggers are
not at all the right solution for it.
2. it relies on the packager guessing the right places to put his
files to trigger processing. The logic should not be "I have an info
file, let's put it here so rpm guesses it's an info file and does the
right thing".
There's hardly guessing involved, you put things where they belong just
like you're currently doing: the canonical location for info files is
%{_infodir} and not %{_libdir}/mypackage/ for example, and the info
trigger would not look anywhere outside %{_infodir}. So for the average
autoconf using software it's taken care of by %configure already.
Again, if something doesn't have a well defined place, file triggers
shouldn't be used.
The logic should be "I tell rpm this is an info file and
rpm does the right thing, including installing it in the right place".
That forces *rpm* to know something about any arbitrary file type and
location you might ever want to handle. You know how "well" that works for
automatic dependency generation - I really doubt you want more of the
same. The knowledge belongs to the packages knowing how to handle
something, be it fontconfig or icon cache or whatever.
This is of course the POW of a packager that has to cope with
imperfect tools that try to be smarter than they can be, not the POW
of the person that writes the smart tools and is convinced he can't do
wrong.
See above, this is why you dont want *rpm* in control of things, only to
provide a mechanism to utilize where it makes sense.
Now, some way to register build-time trigger warnings "your package is
installing X file
Re: File Triggers (was Re: Proposal (and yes, I'm willing to do stuff!): Must Use More Macros)
Le Mer 10 juin 2009 15:29, Panu Matilainen a écrit : > > On Wed, 10 Jun 2009, Nicolas Mailhot wrote: > And this is why the actual script to do whatever magic it > needs to do, when it needs to, would be in a distros fontconfig > package, not rpm. This is totally orthogonal to invoking this script via file triggers or not. > False positive in this context would mean either > a) the cache update run without needing to > b) a package put something into a wrong directory > > a) is harmless as per multiple runs, b) is a grave packaging bug which > with file triggers would be caught when installing the buggy package, What will happen is the very same spec will go bang in one build environment and not another, and people will waste time trying to find out what's different because of the transparent magic processing. That happened many times in the past with the redhat rpm customization that changes rpm behaviour transparently without packager intervention. > instead of next cache update started by something else which then > might blow up/issue warnings long time afterwards. Cache updates triggered by apps are checked upstream. Cache updates triggered by magic rpm transparent rules only happen in a distro environment that uses them. I very much doubt there will ever be a 100% match between the regexps file triggers use to identify files and the rules cache utilities use to identify what to process. > The logic should be "I tell rpm this is an info file and >> rpm does the right thing, including installing it in the right >> place". > > That forces *rpm* to know something about any arbitrary file type and > location you might ever want to handle. That does not force rpm to know anything more than in your proposal. 'Telling rpm X is an info file' can be done via the explicit invocation of %_frob_info_file X, all rpm has to provide is a way for people interested in info files to declare a %_frob_info_file which is then available to other packages (that may use it or not depending on preferences, distro policies and special cases *they* know of). *this* is what is missing today. Packagers know what's in their packages (it's *their* responsability). People know how to write bits of processing appropriate for X or Y content (this is what SIGs and FPC do all the time). People in group 2 know how to communicate with group 1 What's is broken is group 2 can not give group 1 prepackaged routines to use, because rpm does not allow injecting code that spans multiple sections (deps, build, install, post, pre, check etc), and that concern the same subpackage. And thus people have to cut and paste. You can not tell packagers : "To add the font file X.ttf, to your subpackage Y, declare:" %files Y %do_what's_appropriate_for_fonts X.ttf and you're done You have to paste code in build, install, post, preun, etc because of this (and hope you never mess up with the subpackage identifier all those sections expect). Which is a huge PITA and impedance mismatch. The actual file type identification is *not* a problem. It's *less* work than reading the FHS and putting files manually in a place rpm would recognize. And in fact the FHS is not that accurate and for a lot of files location will depend on distro policy, so reading the FHS is not enough anyway. > You know how "well" that works > for > automatic dependency generation - I really doubt you want more of the > same. The knowledge belongs to the packages knowing how to handle > something, be it fontconfig or icon cache or whatever. The processing knowledge does not belong in the package itself. The file identification knowledge is something else entirely. > Well you snipped out the part about "named triggers" which would be > something to this direction: an opt-in feature your package claims > interest in (or "subscribes", whatever terminology you want to use). opt-in in IMHO safer and saner. And more flexible. -- Nicolas Mailhot -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
