Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf
On 7 August 2018 at 09:50, Michael Schroeder wrote:
> On Mon, Aug 06, 2018 at 04:36:07PM +, Zbigniew J??drzejewski-Szmek wrote:
>> this mail is a continuation of an FPC [1] and a FESCo [2] tickets.
>>
>> A proposal was made is to disallow packages in Fedora from using file
>> deps, and to optimize dnf to not load filelists.xml. File deps would
>> still be supported, because external packages and users want to use
>> them, but they would not be allowed for distro packages.
>>
>> Not downloading or loading filelists.xml which are required for file
>> deps would provide significant bandwidth savings (~47 MB compressed)
>> and noticeable runtime savings (~10s at dnf startup) in many common
>> cases.
>>
>> So this is something that is worth exploring, but it's not clear if it
>> is at all feasible.
>
> There's also something that can easily be done and would make
> loading the filelist unneeded in most of the cases: extend the
> primary filelist to include some whitelist of files. The whitelist
> must also be stored in the primary data, so that the solver knows
> what to expect.
That's what Mandrake/Mandriva/Mageia/... has been doing for many
years, there is a small file-deps file containing the ones we end up
generating, mostly from scriptlets IIRC, and we end up with provides
added for those in the main metadata when generating it. Then file
lists are lazily loaded when people want to query them but not used
for dependency resolution.
$ GET
http://ftp.free.fr/mirrors/mageia.org/distrib/cauldron/x86_64/media/media_info/file-deps
/bin/csh
/bin/grep
/bin/perl
/usr/bin/ln
/usr/bin/rm
/sbin/service
/usr/bin/chattr
/usr/bin/guile
/usr/bin/openssl
/usr/bin/pear
/usr/bin/texhash
/usr/bin/tr
/usr/bin/which
/usr/sbin/groupadd
/usr/sbin/groupdel
/usr/sbin/useradd
/usr/sbin/userdel
>> It seems that dnf would need to support loading
>> filelists.xml lazily. In the mailing list discussions, some people
>> said that this would be hard, some people said that it would be
>> possible??? What is the situation here?
>
> Lazy loading of primary extensions is supported in libsolv, the
> demo solver included in the package makes use of that feature.
>
>> IIUC, dnf would need to restart
>> the resolution of a transaction mid-flight once it encounters a file dep,
>> which would require support across the different layers.
>
> No, it works different. At some point the solver creates the ruleset
> needed for dependency resolution. To do this, it has to find which
> packages provide a given dependency. If that's a filename dependency,
> it will check if it matches the default patterns (/etc/* *bin/*
> /usr/lib/sendmail). If it does not match, it will search the filelists.xml
> extension. Here's where libsolv can use a callback to make the lazy
> loading happen.
>
>> If Fedora commits to making use of this, would it be possible to
>> implement this in dnf? What kind of changes would be required?
>>
>> [1] https://pagure.io/packaging-committee/issue/714
>> [2] https://pagure.io/fesco/issue/1955
>
> I don't think this is hard to implement, but there's a little detail
> that needs to be discussed: what should happen if the filelists.xml
> download fails? This can happen because the metadata has been rewritten
> in the meantime. How should the error be propagated back to the user?
>
> Cheers,
> Michael.
>
> --
> Michael Schroeder m...@suse.de
> SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg
> main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}
> ___
> Rpm-ecosystem mailing list
> Rpm-ecosystem@lists.rpm.org
> http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] RFC #417 %optional file attribute
On Mon, 26 Mar 2018, 11:44 Florian Festi, wrote: > Hi! > > We are currently pondering about #417 [1]. For adding a %optional file > attribute that would allow adding file to to %files sections that may > not be built under some circumstances (e.g. some architectures). > > It is already perfectly legal to have files not listed explicitly if > they are within a directory that is included in the %file section. But > some packages (examples wanted) may have trouble using this due to the > way the package files are laid out. > > Otoh %optional would be another spec language key word that packagers > have to deal with and we as RPM upstream developers have a hard time > judging whether the benefit of the attribute really out weights the cost > of bloating the spec language. > > Any input - especially with real world packages that would benefit such > an addition - is welcome. > I personally don't remember missing it in 17 years creating rpm packages and would suggest against using it in distro policy even if it was available. If one file depends on a feature available only on some architectures for example, using it would mean the packager not noticing that it got disabled on another arch where it is supposed to exist. I prefer always being explicit and having explicit conditions defining in which case the file should be there. Florian > > > [1] https://github.com/rpm-software-management/rpm/pull/417 > > > ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Fwd: [Rpm-maint] Fixing macro scoping
On 3 February 2017 at 12:14, Panu Matilainen wrote:
>
> Stunned silence on rpm-maint, forwarding to rpm-ecosystem in hopes of a
> larger and livelier audience...
>
> - Panu -
>
> Forwarded Message
> Subject: [Rpm-maint] Fixing macro scoping
> Date: Mon, 23 Jan 2017 12:30:21 +0200
> From: Panu Matilainen
> To: rpm-ma...@lists.rpm.org
>
>
> Consider the following snippet, originating from
> https://bugzilla.redhat.com/show_bug.cgi?id=552944:
>
> %{!?foo: %define foo bar}
> %define dofoo() true
> echo 1: %{foo}
> %dofoo
> echo 2: %{foo}
>
> I'd assume everybody agrees both %{foo}'s should evaluate to the same value,
> but that is not the case currently. Using a cli-variant of the above:
>
> [pmatilai@sopuli rpm]$ rpm --define 'dofoo() true' --eval '%{!?foo: %define
> foo bar}' --eval '%foo' --eval '%dofoo' --eval '%foo'
> warning: Macro %foo defined but not used within scope
>
> bar
> true
> %foo
>
> The flaw here is that rpm supposedly implements block level scoping for
> macros (so in the above example, "foo" would only exist in the {!?foo:...}
> block), but doesn't actually enforce that, unless a parametric macro is
> "called". Current rpm warns about it, but warnings or not this behavior
> doesn't make the slightest sense.
>
> The question is, what do you think %{foo} should evaluate to in this case?
>
> Fixing it to honor the strict "block scoping" concept is not hard, now that
> the scoping level is honored from Lua too (see
> https://github.com/rpm-software-management/rpm/commit/1767bc4fd82bfacee622e698f9f0ae42c02126fa).
> In this case the above reproducer would emit
>
> %foo
> true
> %foo
>
> Another option is slightly changing the whole scoping notion: parametric
> macros require locally scoped macros for the automatic argument macros like
> %#, %* and %1 anyway (it's flawed too currently, see below). So perhaps the
> macro scoping should follow the current "call level", ie a macro defined
> inside a parametric macro body is local to that macro, and everything else
> is global. In this case the reproducer would emit
> bar
> true
> bar
My first reaction was to prefer this one because '%{!?foo: %define foo
bar} has been used in tens of thousands of packages and sort of worked
"forever"
But replacing define with global is not very hard so no strong opinion...
> I have implementations for both and also a personal opinion, but I'd like to
> hear what others think.
>
> The related flaw is whether locally scoped macros should be visible to
> deeper nesting levels. Currently everything is, including those automatic
> macros:
>
> $ rpm --define '%bar() Bar %#: %{?1} %{?2}' --define '%foo() Foo %#: %{?1}
> %{?2} %bar a' --eval '%foo 1 2'
> Foo 2: 1 2 Bar 1: a 2
>
> I'd consider this a bug, %2 should not be defined within %bar() since it did
> not receive two arguments. So IMO the correct output in the above should be:
> Foo 2: 1 2 Bar 1: a
Yes that's indeed very unexpected as I have always considered %n to be
referring to the parameters of current macros
All %n > %# should be cleared at the same time %# is set
> But what about a macro manually %define'd within %foo() - should that be
> visible in %bar()?
I would expect so
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] rpm -q --whatrequires and rich deps
On 7 April 2016 at 11:49, Miroslav Suchý wrote: > Dne 7.4.2016 v 11:17 Michael Mraka napsal(a): > >> How does this work in other distributions? >> >> I'd like to hear your unbiased opinion that's why I don't include >> neither my preferences nor current rpm behavior for now. > > In rpm I would like to have semantic "which package will stop working if I > remove this package". I.e. when I run: > rpm -e foo > then I will get some errors that foo cannot be removed because A,B and C > requires it. I would expect that --whatrequires > gives me the same list (sans transitive requires). > > So in your example: >> An example to think about - have a package with following requires installed >> richdep.spec: >> Requires: A >> Requires: B >> Requires: (C and D) >> Requires: (E or F) >> Requires: (G if H else I) >> >> Which of the following queries should include 'richdep' in the output? >> rpm -q --whatrequires A >> rpm -q --whatrequires B > > richdep for both above. > >> rpm -q --whatrequires C >> rpm -q --whatrequires D >> rpm -q --whatrequires '(C and D)' > > richdep for all three > >> rpm -q --whatrequires E >> rpm -q --whatrequires F > > none What if only one of them is currently installed? >> rpm -q --whatrequires '(E or F)' > > richdep > >> rpm -q --whatrequires G >> rpm -q --whatrequires '(G if H)' >> rpm -q --whatrequires '(G if H else I)' > > for all three: richdep if *I* have H installed, otherwise none > >> rpm -q --whatrequires H > > none > >> rpm -q --whatrequires I > > richdep if there is no H installed on my machine, otherwise none > > Just my 2 cents. > > -- > Miroslav Suchy, RHCA > Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys > ___ > Rpm-ecosystem mailing list > Rpm-ecosystem@lists.rpm.org > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Using rpm db to track unneeded packages
On 1 April 2016 at 13:05, Tomas Chvatal wrote: > Hello, > > I was wondering if it would be possible to extend rpmdb to contain > information about how package was pulled in the dependency graph. > > At this point we at openSUSE have some sort of solver trying to > magically do it in zypper and you guys at Fedora have it in dnf. > > I would more like to see it tracked in the rpm because that way we > could properly have all packages in the db, including the ones users > installed via rpm commands and could see if they are > a) directly requested by user > b) just dependency of something and thus eligible for removal urpmi has its own tracking (a simple text file /var/lib/rpm/installed-through-deps.list) but I think that would be nice to have it in rpm directly so that it works whatever tool is used to install the packages > I guess dnf/zypper would just flag them during install as True/False > depending if they are direct request or dependency and for rpm we would > always flag them as True for the solver as requested by user. > > Would something like this make sense? > > Cheers > > Tom > ___ > Rpm-ecosystem mailing list > Rpm-ecosystem@lists.rpm.org > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem > ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
