Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
An entire *.rpm file needs to be read only because rpm-metadata chose the file digest to be included in metadata. There is no way for a plaintext file to contain its own digest, true for zchunk as well as current rpm format. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
Um, rpm headers are not compressed, so there is no decompression overhead retrieving a tag value, and there is no reason why an entire rpm needs to be decompressed in order to retrieve a value. Presumably you are obliquely referring to deltarpm creation, not rpm tag value access. After watching the video, I am also unsure how you got the impression that the number one request was for faster access to fields in an rpm header. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: zram-generator
Why not Haskell? Seriously: you provide no reason for rust other than that "C is not attractive". C is surely a reliable implementation language for long running system services. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/UDRQBY42WWBWPRTOE7YJJHIDT2MSZ2NS/
Re: Automating Package Review (Was: fedora-review -- do we have a maintainer?)
To answer your question solely because I don't like FUD driven phears monger int discussions RPM based depsolvers select packages based on heuristics, including what is already installed. Any malicious package that had Provides: glibc would most likely be ignored because glibc is already installed. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/5OIFLSJAWO5OTVHNPAT62Z32IWE3BOG2/
Re: fedora-messaging for Copr - what do you need?
There is no reason why a connection to an AMQP broker cannot be opened and carried into a chroot that has networking disabled: rpm installs already perform chroot's carrying multiple open Berkeley DB fdnos into the chroot. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/DFDYXMSQ55CS6MFUNQOYQ47LJZE7VJMH/
Re: fedora-messaging for Copr - what do you need?
Changing Fedora infrastructure to use AMQP brings up a different usage case. Build systems tend to eventually invoke rpmbuild to do the actual build. Would you like rpmbuild to, say, send time stamped progress messages directly through AMQP? The usage case for AMQP messages in rpmbuild starts with the start/end times to collect statistics on package builds, but could also include other resource usages needed by rpmbuild. With RPM+AMQP, there is additional detailed information about installations that can be sent directly as well. Any interest in receiving AMQP messages directly from RPM? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/KSAKHYHPVQVHWJFMEBSL6IF3W2JX6MDM/
Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf
There is a further optimization for file dependencies not in some whitelist that might be undertaken by depsolvers like dnf: lookup the path in the already installed packages. The fundamental need for depsolvers resolving a dependency is the ability to traverse a mapping from path to package in order to determine whether there is a "newer" package available so that the package that provides a path can be upgraded. Resolving the path dynamically to a package name, or even using the existence of a file as a means to resolve a dependency, is a better approach than downloading megabytes of data "just in case". ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/KK5OTIW6WWJ43IXISNYMWMGBYK4FLSZ7/
Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf
Perhaps you do not understand the implications of "maintained" wrto a whitelist. A cross repository file dependency should lead to a request to add a path or pattern to a whitelist for the other repository to provide a file path, not to a lazy download of megabytes of mostly unused data "just in case". ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/GAM3ZEK7YS5KEHMDZXM437GKZS4UQ2LJ/
Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf
My issue is the misdirection discussing lazy filelist downloading as a "solution" to the "problem" of huge amounts of data that is forced to be downloaded and loaded. The issue has been discussed repeatedly without a solution. Adding -- and maintaining -- patterns or whitelist exceptions, which moves file dependencies into primary.xml is actually an approach that solves the problem, and scales to multiple repos as well, each of which also will have their own patterns and whitelist. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/C423LKM74ZVLECWUR7VFX5CZIMEBUAJW/
Re: [Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf
> On Tue, Aug 07, 2018 at 08:50:25AM +, Michael Schroeder wrote: > > Yep, that sounds like an excellent idea. > > > That's a good question. The time window where this can happen is not > be that big, because without filelists the loading of metadata is > quicker. But it's still non-zero, so given enough machines and enough > updates, it'll be hit occasionally. > "Occasionally" is a pleasant euphemism. > A dirty solution would be to simply error out, not nice. > > I think the best solution if part of the meta-data cannot be > downloaded, is to restart the download of metadata in non-lazy mode. In > other words, if the lazy approach fails, repeat the process exactly > like it is done today. > There is no other solution to conflicting tasks accessing identical paths. Well you could put a file with a time stamp (to be able to detect stale)in the same directory, and then teach down loaders to honor the lock, or rename a directory in place after updating, but both are tricky to implement with rsync mirroring. > One thing that mitigates this issue is that we have multiple mirrors, > and they cannot be all updated at the same time, so some mirrors will > carry "stale" metadata, and dnf should be able to hit some other mirror > that still has the old filelists. Thus, I think it should be OK to start > with the "dirty solution", if implementing the fallback is complicated, > and implement the fallback later. > One could also map the filelist lookup into a http based service instead of downloading the entire list, lazily or not, with compression or not, with client cache management or not. Rsync mirroring of package metadata is very last century. But "occasionally" downloading file lists does not solve the problems stated, either then, or now. Till next time the issue is discussed ... 73 de Jeff > Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/R4UDGQY4P4AOTQEQ226HT2Z6ON2MNS2X/
Re: lazy loading of filelists.xml to speed up dnf
Sure extending the whitelist -- either through hard coded patterns or an actual list -- obviates the need for downloading bloated file lists. Those techniques simply move "useful" data out of file lists and into another file, either primary.xml or a whitelist. Patterns (and Provides: /path) has already been tried, and adding another file to distribute and download just repeats an implementation that is known to have issues. Lazily downloading additional info will not prevent file lists from being downloaded when needed. The problem is only partially solved. Or just ban file dependencies everywhere: that simplifies a huge amount of code in rpm and depsolvers that can then be deprecated and removed if file dependencies are forbidden. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/KIQS2YCNNHZ75MVFMQLF4MUNYKD4OB77/
Re: lazy loading of filelists.xml to speed up dnf
At the risk of again being banned from Fedora, I point out this thread from 2011, discussing/measuring various techniques for reducing the size of metadata containing many file paths for Mandriva http://rpm5.org/community/rpm-devel/.html ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/R2DWTHETJKY3GF7IW2MZCUIAQEYU7HGP/
Re: lazy loading of filelists.xml to speed up dnf
Bloom filters have false positives that can lead to difficult to predict behavior. The core proble(s) being reported with file dependencies is the size of the download, the cost of the load, and the size/number of copies one for each location managed by dnf. These are all problems that are straight forward to mitigate without forbidding file dependencies. (aside) I personally like Bloom filters, and have used them successfully multiple places in rpm code. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/DQX3EXZD54SRIDVF3WWP5WT735KQEYCF/
Re: lazy loading of filelists.xml to speed up dnf
Whether dnf does what yum does misses the point: Lazy downloading of file dependencies did not work then and does not work now. Here's why: The fundamental decision made by depsolvers is what packages are "newer" and need to be upgraded. When _ANY_ commonly used package has a file dependency outside of the paths chosen to be delivered in primary.xml, then _ALL_ file dependencies need to be downloaded in order to find the package that contains the path so that the upgrade decision based on "newer" can be determined by the depsolver. Without explicit packaging policy -- and tools to verify -- forbidding all file dependencies outside of the magical paths preselected to be included in primary.xml, then filelist.xml will surely **AGAIN** be downloaded **ALL** the time. Since every depsolver has chosen to implement the magical paths in code rather than sharing a list of file patterns permitted somehow, then this cycle will be repeated again and again. The new usage case of /usr/libexec, and the older usage case of /bin vs /usr/bin indicates that history is repeating itself. TL;DR Magic file path patterns implemented in code and lazy loading of filelist.xml "just in case" will never succeed. Even if you Get It Right! for Fedora (and CentOS) packaging policy, there will be issues with custom 3rd party and end-user repositories that do not conform to the magical paths chosen. It is also very not surprising that there are very few file dependencies remaining in Fedora after several jihads (and policy) to eliminate file dependencies in Fedora packages. Another jihad (and tighter policy) cannot solve the problem of custom non-policy-conformance in custom repositories, no matter what. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/KAOMH4Z2NMRZKIMXYP7DYBY3B7BN3AVM/
Re: %{valgrind_arches}
Try ... %check %ifarch ppc64 ppc64p7 exit 0 %endif ... My comments apply to the rest of what you appear to be proposing everywhere. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/BF5MQ4HNFZPYAJC7MAC7EMB2LNAH7PG4/
Re: %{valgrind_arches}
Yuck. So you are recommending using 14 lines (with comments) of spec file goop that uses 2 %ifarch build section tests in order to set/unset a macro. There's further baggage in spec files needed to add a BR, pass an option to configure, add libraries to link, etc You are in the wrong namespace testing arch in spec files to determine +/- valgrind. To illustrate what I am saying, consider endianness. There was a lot of code that used to test the vendor "sun" or the platform "VAX" in order to determine endianness, all of which had to be patched when circumstances changed and Solaris was released for ix86. Your 14 line example will suffer a similar fate if valgrind is fixed to run on ppc64, or ppc64 aliases start to proliferate. There is a need to refactor all of this framework to simplify building packages with or without a subcomponent like valgrind that does not use explicit test on arch to imply +/- valgrind subcomponent. Gentoo manages this quite well with XML and inheritance instead of explicit logic. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/3KXHXM5QLL7NOH7BUJTLA5DU5GWEIMC7/
Re: dnf history - change in how rpmdb checksum is computed
This simply is not true. Whatever "rpm format" means, historically RPM itself has always gone to some lengths not to expose E: to users to simplify the endless fog of dependency hell clutter. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/XHYTGM2L7PNOWNYKA6T26B7ACDGMX3DD/
Re: Making Fedora secure - Package exit policy for security
Don't rely on MODERATE or LOW distinctions to drop a package in FX+2. Just drop all unfixed packages with the same policy. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/EOQ3DJKOMBGUHDQ66XTL43RZCSV4MAFN/
Re: tool to order packages by build dependencies (rpmbuild-order)
You are absolutely correct: there are times, e.g., that drinking coffee is "more important" to me than say, replying to e-mail. However I don't think that Haskel is the best language to feature up an already existing uglix tool. Meanwhile I do not know of a tool that schedules rpm builds efficiently. I'd be happy to be wrong. See the build scheduler for the Hurd in NixOS for what I was referring to http://hubble.gforge.inria.fr/hubble.html ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/HM64XDUEBJJPXEG43DUWIQVSL5WJVQHA/
Re: tool to order packages by build dependencies (rpmbuild-order)
There are easier ways to accomplish build ordering using tsort(1) and a minor amount of scripting with the same simplifying assumptions your program is making: 1) no cycles 2) every package provides its own name 3) every package supplies prerequisite package names without versions tsort(1) just needs 1 line of input for each package containing the name followed by a list of prerequisites ending with a new line More important than build ordering is scheduling a set of builds efficiently across multiple build machines to complete in the shortest amount of time taking into account the cost in memory and cpu time of each package build. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/NSQZKDOHSTZLI2LJXTLIVJRFKAZT4VEM/
Re: dnf history - change in how rpmdb checksum is computed
This thread was about compatibility between dnf and yum: rpm itself has no usage case identifying whether an rpmdb has been changed. But you are correct that installation of a package by any tool -- including dnf and rpm atm -- needlessly causes yum to warn that the rpmdb has changed which is rather silly IMHO. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/ZMU4CE2ZSHOSQCCG7AW6FXREDGCU3MNF/
Re: dnf history - change in how rpmdb checksum is computed
The real problem is that both E:N-V-R.A and N-E:V-R.A are equally imprecise. The concept of "reproducible builds/installs" requires much more complete identifiers for serious work. But that was not the question asked in this thread. So calculating both checksums, on rearranged plaintext items, for compatibility, kinda misses the underlying need to verify system installs on hundreds of machines. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/G5PQ6DZKLVJJYPJCQG2VVQQMRAITETJ3/
Re: Fedora crda To wireless-regdb Upgrade
Package renaming using Obsoletes/Provides will be a problem *ONLY* if
applications confuse the version of the package with the CRDB format version:
these are intrinsically different versions even if there is a single package
renaming event that is coincident with the CRDB format change.
Personally, I would add a CRDB format tracking version as a virtual Provides:
CRDB-N = X:Y-Z where. all of {CRDB-N, X, Y, Z} are merely placeholders for
whatever values seem natural and "appropriate". YMMV.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QELITCW6YH4EKS2HNPXZHNBFQRADZL57/
Re: dnf history - change in how rpmdb checksum is computed
Use headerFormat() with a configurable format string to extract the package identifier item in the list that is check summed and have it both ways. Why anyone wishes to preserve compatibility with yum's bloated history database in order to flip between two depsolvers is left to RH TAM's to explain to customers. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/CT4KBPW6YAP6O6BJNU4UQQSHKGCR4I3Q/
Re: Release criteria proposal: installing / removing software
There is also historical precedent in open source software for using the terms "recommended" and "suggested" rather than "appropriate". Installing/upgrading licensed software is never appropriate. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/NYYA36BKNHCAR35DU2Y54WNL67GLXZTF/
Re: Release criteria proposal: installing / removing software
Try "expected" rather than "appropriately" ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/FBM3HOWXN7RF5LJ4M6J2HWESB3UT3L2H/
