Re: RPM name collisions

[Daniel Mach]

On 5/5/21 7:08 PM, Neal Gompa wrote:

On Wed, May 5, 2021 at 11:55 AM Qiyu Yan  wrote:


在 2021-05-05星期三的 07:44 +0200，Dan Čermák写道：

przemek klosowski via devel  writes:


Is that something we need to worry about? I couldn't think of any new
rules to impose on repositories, but maybe dnf should have more
explicit
warnings when it sees multiple versions of the same package, or at
least
a way to show such versions.


Or how about teaching dnf that only certain repositories are allowed to
be used for updates (with an allowedlist for exceptions)? Then
microsoft
or any other third party repo could put hello-5000-1 into their repo
and
it could never compromise your system, as dnf would not consider the
3rd
party repo a valid update repo for a base system package.


Neal, I remember you had a really good idea about this - repo stickiness 
(+repo equivalence classes). It might be a better option than vendor 
stickiness. I'd like to explore this more in DNF 5 (see my excuses why I 
don't want to do it in DNF 4 below).





That would require dnf to track where it got the package from though
and I am not sure if it does that at the moment?

This reminds me of an idea named Vendor Change from Zypper of openSUSE
https://en.opensuse.org/SDB:Vendor_change_update
This approach seems to solve our problems here?


Well, we do have the sticky vendor feature in DNF, DNF on openSUSE has
it switched on by default[1]. If we want to have this feature turned
on in Fedora, we could look at having it switched on.

[1]: 
https://build.opensuse.org/package/view_file/openSUSE:Factory/libdnf/libdnf-0.55.0-Switch-allow_vendor_change-off.patch?expand=1



Yes, there's basic support for ventor stickiness in DNF.
It probably works fine with dnf install and upgrade.

I have concerns about the following commands:

* downgrade
  * not supported in Zypper
  * no support in libsolv
  * dnf emulates downgrades with installs, that implies vendor changes
* distro-sync
  * unsupported with allow_vendor_change=1
* install
  * installing explicit NEVRA lead to vendor change
  * vendor changes are not displayed in the transaction table
* updateinfo
  * dnf probably considers all package versions as update candidates
  * needs thorough testing at least
* literally all dnf commands that query the package set,
  because vendor impacts the meaning of the latest package.
  * list
  * info
  * repoquery
  * repoclosure
* COPR
  * enabling a COPR repo would do nothing if the repo provides 
alternatives to packages shipped in Fedora

  * maybe a good thing, but definitely a change in user expectations

What's also important, DNF team currently doesn't have capacity to fix 
all new bugs related to turning `allow_vendor_change` option on. Our 
priority is RHEL 9 (that includes new modularity depsolver among other 
bugs/RFEs) and DNF 5 (doing it right takes time and is relatively hard, 
but we believe it pays off long-term due to bringing better APIs, 
overall consistency etc.)


To be clear, I like the idea of vendor/repo stickiness turned on by 
default, it could solve a lot of problems and eventually provide a very 
simple alternative to modularity. I'm just not convinced that the DNF 
stack is ready for that now and turning it on would most likely make 
user experience worse. I'm fine with my team looking into this when we 
have capacity to do so - that means in about one year from now.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: bootstrapping without .spec modification

[Daniel Mach]

On 1/29/21 12:44 PM, Miro Hrončok wrote:

On 29. 01. 21 12:25, Zbigniew Jędrzejewski-Szmek wrote:

Hello fellow packagers!

The subject of bootstrapping came up on fedora-devel recently.
I had the following idea, about which I would love to hear some feedback:

== Problem:
building packages with bootstrap currently involves doing *two*
patches to the spec file: first to add '%global _without_bootstrap 1',
then comes a rebuild, second to remove the macro, and then comes
another rebuild.

== Partial solution
Let's have an rpm that provides a single file that sets the macro for us:
$ rpm -qpl noarch/rpm-with-bootstrap-0-1.fc34.noarch.rpm
/usr/lib/rpm/macros.d/macros.rpm-with-bootstrap
$ cat rpm-macros-bootstrap/macros.rpm-with-bootstrap
# Enable %%with_bootstrap for all builds
%_with_bootstrap 1

Then we can do the following:
$ rpmdev-bumpspec 'Do rebuild w/ bootstrap'
$ mock -i rpm-with-bootstrap
$ fedpkg mockbuild
$ rpmdev-bumpspec 'Do rebuild w/o bootstrap'
$ mock -i rpm-without-bootstrap [3]
$ fedpkg mockbuild
Voilà!

The same pattern should work with side-tags in koji.

I prepared a poc implementation in [1] which builds
rpm-{with,without}-{bootstrap,tests,lto}, and a test package [2] which
prints the values of %with_bootstrap, %with_tests, %with_lto.

== Full solution
If we have automatic version bumps, this would become even simpler:
$ mock -i rpm-with-bootstrap
$ fedpkg mockbuild
$ mock --dnf-cmd remove rpm-with-bootstrap
$ fedpkg mockbuild


A more general note: Your examples involve mock, not Koji.

For mock, you can already do:

  $ fedpkg mockbuild --with bootstrap
  $ mock --install 
  $ fedpkg mockbuild

For Koji, you cannot install arbitrary packages.

What if instead, Koji allowed to set arbitrary macros on builds (and it 
keeps their definition for further reference). That way, you will be 
able to do:


  $ fedpkg build --with bootstrap
  $ koji wait-repo ...
  $ fedpkg build

 From the same commit. No package installation required.



Yes, defining rpm macros in koji would solve that for the build system.
I'd love to have the following features:
* general macros that are not package or arch specific (probably most of 
them)

* per-(package,arch) macro overrides
* macro inheritance across koji tags
* don't forget that NVR has to be unique in koji so you can't build the 
same build twice. Having an ability to set %dist via koji might be nice 
for bootstrapping.


The downside is that anyone that is not using koji would have to 
retrieve the macros from somewhere (export macros into a RPM file and 
ship it as part of the release?). But that's only if the macros are used 
for building production RPMs that land in Fedora repos. RPMs used for 
bootstrapping don't have to be rebuildable outside the build system.



It seems to be related to this:
RFC: Feature macros (aka USE flags)
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/4DWDPKYBOXTCGKXJTOIVPO34A5BTOE3T/


And it's also not far from defining build macros in modules:
https://docs.fedoraproject.org/en-US/modularity/building-modules/fedora/defining-modules/#_build_macros_optional
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Software Management (RPM, DNF) 2020 retrospective

[Daniel Mach]

Dne 18. 01. 21 v 11:29 Kamil Paral napsal(a):
On Mon, Jan 18, 2021 at 9:36 AM Daniel Mach <mailto:dm...@redhat.com>> wrote:


$ dnf offline-upgrade
$ dnf offline-distrosync
    * New commands to upgrade your system on reboot


Sounds great. But I don't see those commands neither in F33 nor in 
Rawhide. Am I looking wrong? Thanks.




They're not distributed as RPMs, they're part of 
python3-dnf-plugin-system-upgrade. We should probably make their 
presence more obvious.


Created https://bugzilla.redhat.com/show_bug.cgi?id=1917378
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Software Management (RPM, DNF) 2020 retrospective

[Daniel Mach]

Hi,
On behalf of RPM and DNF teams, I would like to highlight changes that 
have appeared in our packages in 2020. Thanks everyone for your bug 
reports and patches!



RPM
---

$ rpm --eval '%[0 < 1 ? "true" : "false"]'
  * Support for complex expressions and conditionals with short-circuit 
logic etc in macros via %[...] and %{expr:...}


$ rpm --eval '%[v"1.10" > v"1.2"]'
  * Rpm versions as native objects in expressions, eg v"3.1-1"

$ rpmbuild
  * Parametric macro dependency generators offer a lightning fast way 
to generate dependencies where applicable
  * Meta dependencies, eg Requires(meta) allow specifying dependencies 
without introducing ordering constraints

  * Dependency generation based on MIME types

$ rpm --rebuilddb
  * New sqlite3 backend
* More robust than the Berkley DB backend
* SQLite3 has also a more permissive license than the latest BDB
  * NDB backend promoted to stable
  * Support for migration (reading) from BDB databases without linking 
to BDB



DNF
---

$ dnf needs-restarting
  * Introduced the needs-restarting.d drop-in directory

$ dnf reposync
  * Fixed modularity issues, able to sync modular repos correctly

$ dnf groups-manager
  * yum-groups-manager is back
  * Upstream only, to be released in Fedora soon
  * https://github.com/rpm-software-management/dnf-plugins-core/pull/412

$ dnf module switch-to
  * Switches the module stream and also performs distro-sync on the new 
stream content


$ dnf history store/replay
  * New commands for storing a transaction and replaying it on another 
system


$ dnf system-upgrade
  * The code was refactored to use the store/replay feature in the 
background

  * Landing in Fedora now

$ dnf system-upgrade
  * Comps groups are upgraded during the system upgrades

$ dnf offline-upgrade
$ dnf offline-distrosync
  * New commands to upgrade your system on reboot


MICRODNF


$ microdnf module 
  * New commands for managing modules

$ microdnf --config
  * Microdnf reads values from dnf.conf
  * It is possible to specify a path to the config

$ microdnf --refresh
  * Force repodata refresh
  * Microdnf also honors the metadata_expire configuration option


LIBDNF
--
  * Many changes to make the DNF features available in microdnf and 
PackageKit (and Gnome Software)



DNF 5
-
  * The goal is to remove redundant code and make sure all tools built 
on top libdnf work the same (DNF currently uses a different code path 
than microdnf and PackageKit)
  * We're behind the plan because we spent the time fixing DNF 4 stack 
instead (modularity, new features, crashes, etc.)

  * There's a COPR repo with unstable nightlies:
* dnf copr enable rpmsoftwaremanagement/dnf5-unstable
* dnf install microdnf5
* dnf install python3-libdnf5 perl5-libdnf ruby-libdnf
* WARNING: Here be dragons! The software is not ready for general 
use yet.
  * The next steps are: finish the core functionality, stabilize the 
API and make the package available in Fedora in parallel with the DNF 4 
stack

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Delta RPMs in Fedora 34

[Daniel Mach]

Dne 05. 01. 21 v 0:50 Kevin Fenzi napsal(a):

On Mon, Jan 04, 2021 at 06:29:13PM -0500, Matthew Miller wrote:

On Mon, Jan 04, 2021 at 10:21:15PM +, Matthew Almond via devel wrote:

There's been a lot of interesting talk about the state and future of
drpm. I'd like to propose we continue the conversation about that with
a different subject line :)


Okay, fair. I have a proposal.

Right now, the problem is that making delta rpms is expensive, and therefore
we aren't making very many, which makes them even less useful. Plus, we're
only making them between updates and for packages where those updates are
frequent, that means you need to keep on top of things, which may be best
practice but is most difficult for low-bandwidth users who might most
benefit in the first place.

So, the first thing we need to do to fix this is move deltarpm creation out
of the updates process. Kevin Fenzi tells me this would mean we'd need a
separate delta RPMs repo, which doesn't sound like a bad thing to me, but
we're not sure offhand if DNF can handle that without modification.


Yeah, I don't recall how dnf looks for drpms.
Right now they are in the same repo, using the same repodata.

If we moved them to a new repo would they get found correctly?


This would let us make the delta RPMs asynchronously and not block updates.
And, it would also give us the ability to roughly see how important they are
to users, because we could see how popular that repository is compared to
the updates repo.

I also remember when this was a killer feature for Fedora, and without any
real way of judging use and demand, I'm hesitant to kill it off. But that's
definitely plan B. We can point people who are in low-bandwidth situations
at Silverblue, CoreOS, and Kinoite as the preferred approach.


Yeah, I came up with one more possible way we could get more drpms with
our current setup, but need to talk to pungi maintainers and see if it's
doable. :) After that, it's either split things out or drop drpms I
think.



To be honest, I don't understand why drpms are related to Pungi at all.

Deltas are optional, if they're not available, a normal RPM is used.
They can be processed asynchronously (as mentioned earlier in this 
thread) and injected in repos once they're ready.


Please note that we're talking about 74 drpms in F33.x86_64 updates repo:
http://ftp.fi.muni.cz/pub/linux/fedora/linux/updates/33/Everything/x86_64/drpms/

Sometimes I'm wondering if it's worth it and if Fedora shouldn't move 
away from drpms.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)

[Daniel Mach]

Dne 24. 12. 20 v 22:54 Matthew Almond via devel napsal(a):

Depends on how it got there, and what you asked for. Here's some
examples:

1. cp foo.rpm /var/cache/dnf//Packages/ && dnf install foo
...will fail the librepo full file check, and it'll be re-
downloaded.
2. dnf install /root/foo.rpm || rpm -i /root/foo.rpm
(not actually tested) will likely fail with CPIO/payload error

Note that tools like rpm2cpio and rpm2archive will also fail on
transcoded rpms. I have an open task to make the dnf plugin not
transcode with 'yumdownloader' or 'dnf download' (plugin) as those are
reasonable command to run. I will look at making error messages better
and/or making some of these use cases work.



This concerns us (speaking for RPM and DNF people) a little bit.
If the transcoded RPM cannot be used as a regular RPM, it probably 
should have a different identity, for example a different suffix than 
.rpm. RPM and DNF are designed for generic use cases. I see these 
transcoded packages as a "cache" tailored for btrfs based systems only. 
It would be probably good to draw a border between them.


If I understand it correctly, the transcoding happens on each host. Have 
you considered transcoding all RPMs in a repo on server instead? Or 
would that be inefficient and increase network traffic too much?

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: EOL and Obsoletes in Modularity

[Daniel Mach]

Dne 03. 11. 20 v 13:14 Petr Pisar napsal(a):

On Tue, Nov 03, 2020 at 12:26:46AM +0100, Dan Čermák wrote:

Daniel Mach  writes:

The API is clear: DNF expects additional 'modulemd-obsoletes' YAML
documents in modules.yaml. The new document format is getting into
libmodulemd and there's going to be documentation on how to write it.

Then there's a question how to get the documents into modules.yaml. From
my perspective, it's up to Fedora infra/releng/packaging people. Whether
it should be in dist-git, git repo (similar to modulemd-defaults or
comps), PDC, Bodhi (similar to updateinfo) or somewhere else - that's
entirely their call.


I think the package/module maintainer should be able to set the EOL and
obsoletes for their module in a simple fashion, preferably without
requiring to create tickets for releng or anyone else. So that leaves
dist-git as the only option, right?


I also like the idea of having EOL data in dist-git in hands of the packagers.

The problem is that a compose process run by relengs does not use dist-git at
all. It picks up builds from Koji and data from various git repositories. The
various git repositores is where the module default profiles live. Problem is
that the various repositories are not dist-git. They have a rigid commit
access by a few people. Those are not packagers.

We could create a dedicated repository in dist-git for collecting the EOL data
from various module maintainers and give the module maintainers a commit
access there. But the problem with dist-git is that a special repository does
not fit into the namespace schemata (rpms, modules, tests).

We could create a dedicated repository on pagure.io instead, give maintainers
a commit access there (the account system is shared with Fedora, I think) and
confugure pungi (a tool run by relengs to make composes) to read EOL data from
there. But there still had to be a manual step of granting the commit access.

Another option would be adding the EOL data into modules dist-git
repositories. To the same branches where modules are built from. But
a different file. Pungi would enumarate all module builds directed to
a compose, locate their dist-git sources by mapping name:stream to
git://src.fedoraproject/org/modules/name.git#stream and download EOL data from
there. That would enable module maintains to control EOLs simply by pushing an
eol.yaml file into the branch of the module.

Wouldn't relengs object that it's not accountable because the EOL data can be
updated asynchrously of module builds and tagging in Koji? And isn't that what
we want to acheive -- EOL data independent from the module builds?

Alternatively we could append EOL YAML document to modulemd YAML document in
the dist-git (a YAML file can contain multiple YAML documents) and rebuild the
build. That way the EOL data would get together with the modulmd to Koji
and to the compose. But that would bind the EOL data to module builds. If we
wanted to EOL a module stream, we would have to make an unnecessary module
build. Would it be still worthwile?

-- Petr


Thanks for your write-up, it perfectly describes the workflow issues and 
challenges.


The mapping from dist-git to compose or bodhi update is not entirely 
straightforward. It starts when a package is built and tagged in a tag 
associated with a release. That makes me think that extending this to 
modulemd-obsoletes could also work, but that would require "building" (a 
simple import from dist-git to the build system might be sufficient) and 
tagging them in the build system. That would allow tagging the 
modulemd-obsoletes for a compose or attaching them to a bodhi update. 
There's only one if - if someone could extend the infra tools to support 
this new content...


The non-dist-git git repos with limited access are a workaround for lack 
of tagging support - you simply export the repo content and include it 
where you need without tagging / cherry-picking / organizing content in 
the build system.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: making download of filelists.xml(.zck) trigger on demand

[Daniel Mach]

Lazy file list loading? Yes, that's on DNF's TODO list already, but (to 
be honest) not on top - there's always something more important. It's 
not getting into DNF4, but it may get into DNF5 later on.



Dne 02. 11. 20 v 20:13 clime napsal(a):

On Mon, 2 Nov 2020 at 18:55, Vít Ondruch  wrote:



Dne 01. 11. 20 v 11:58 clime napsal(a):

Hello!

First of all, I don't really know what I am talking about here but I
noticed the `dnf update` operation downloads among other things
`filelists.xml` (optionally compressed by zchunk, thanks Jonathan
Dieter!!!) and I remember there was a thread on devel mailing quite
some time ago by Zbyszek from which I understood this data is only
needed when filesystem paths are used as package requirements or when
a fs path is specified as direct argument to `dnf install`.



This was the case for Yum, but have never the case for DNF since its
beginning. I think there were some reasons, such as that the solver in
case it finds it needs them would need to stop, download the data and
start from beginning. Other reason could be that nobody optimized it yet.

Anyway, I guess you'll find more info in Bugzilla.


Hello vit, thank you very much, I will try to look it up. M.




V.



   Could we
then, please, trigger download of this file only if and when needed
and not sooner? It seems to take more than half of the total download
size e.g. for fedora-32-updates repo so it could improve repodata
download times quite significantly. Again, I don't really know what I
am talking about here but this came to my mind recently so I wanted to
write it down just in case it would be possible :).

Thank you very much
clime
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: EOL and Obsoletes in Modularity

[Daniel Mach]

Hi,


Dne 27. 10. 20 v 13:52 Martin Curlej napsal(a):

Hi,

EOL and Obsoletes were planned as a feature of Modularity. The feature 
should enable to set shorter/longer life cycles on Modules than the OS 
release. The initial idea was to set this information in the disgit 
metadata of a Module. As time went by the requirements have changed.


From DNF team's perspective, a packaging system is not complete without 
Obsoletes. That's why we believe module Obsoletes are a must have to 
ensure smooth system upgrades and regular updates on systems consuming 
Rawhide (or any other rolling stream). I created a Fedora Change[1] 
which got accepted already. We'll deliver support for module Obsoletes 
in DNF as soon as possible and we hope it's going to be used in Fedora 
34 (or 35 at latest).


The API is clear: DNF expects additional 'modulemd-obsoletes' YAML 
documents in modules.yaml. The new document format is getting into 
libmodulemd and there's going to be documentation on how to write it.


Then there's a question how to get the documents into modules.yaml. From 
my perspective, it's up to Fedora infra/releng/packaging people. Whether 
it should be in dist-git, git repo (similar to modulemd-defaults or 
comps), PDC, Bodhi (similar to updateinfo) or somewhere else - that's 
entirely their call.



[1] 
https://fedoraproject.org/w/index.php?title=Changes/Module_Obsoletes_and_EOL


 From my point of view right now it seems that this information should 
not be a part of the module (disgit, repository metadata). It is prone 
to human error when we leave this in the hands of a packager. So this 
would need a review of Engineering to be reliable.


Next is that a lot of 3rd parties like to handle the EOL and Obsoletes 
of packages/modules by other means, which makes this redundant.  Also, 
as the release cycle of Fedora is so fast, I am not sure this is a 
necessary feature at all


I spoke to people in the community and I got mixed information/opinions, 
so I want to open a discussion about this feature. So WDYT? Is this 
necessary for Fedora? If yes, how should we handle this?


--

Martin Curlej

Software Engineer, Product Owner

Modularity

Red Hat



mcur...@redhat.com 




___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Modularity Improvements Objective

[Daniel Mach]

Hello everyone,
There's a Modularity Improvements Objective draft available[1].

The Objective summarizes the work that is in progress already as well as 
highlights our plans for Fedora 34.


We're planning to fix the current modularity in Fedora 33 and 34.
We may look into alternatives or bigger design changes in Fedora 35 and 
later.


You can find more details in the Objective document[1].

- Daniel


[1] https://fedoraproject.org/wiki/Objectives/Modularity_Improvements_2020
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Removing packages from module

[Daniel Mach]

Dne 17. 07. 20 v 23:14 Aleksei Bavshin napsal(a):


Yes, that's something I already accepted.
The real question is how to do the change in f33 considering that f32
and f33 modules must be built from the same modulemd file. I only see
the ability to disable module stream build for f33.
Hmm, it seems this is caused by the dist-git implementation of 
modularity source control management.
There's no name or stream in the yaml file. If I understand it 
correctly, the values are obtained from following places:

name == name of the dist-git repo
stream == name of the dist-git branch

I'll make a note for myself to make the fields mandatory so you can use 
any any branch name. That would allow you maintaining 2 versions of the 
source yaml file and build the modules differently for different platforms.




And now I'm curious what would happen if I specify `platform: [-f33]`
and publish new module build. Would that remove previously published
metadata from rawhide?
I guess the right answer is somewhere around servicelevels and `eol`
specification.

It probably depends on the implementation.
Not sure if Fedora infrastructure releases module for the latest module 
version (which would be none in this case) or the latest available 
(which would be the previous module build).

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Removing packages from module

[Daniel Mach]

Dne 17. 07. 20 v 14:15 Aleksei Bavshin napsal(a):

On 7/17/20 3:27 AM, Daniel Mach wrote:

Dne 17. 07. 20 v 11:04 Miro Hrončok napsal(a):

If the package was part of the module during the release of Fedora 32,
dnf will see 2 versions/builds of the module-stream:

   - the one that existed during the release (containing the package)
   - the one from updates (no longer containing the package)

The "original" version will still filter out the non-modular version
of the package from any transaction.

And given the way modular filtering works, a package cannot
technically be "demodularized" this was during one release, unless you
force a module reset from another package's scriptlet.


Thanks for clarifying this, Miro. So the filter is built based on the
module metadata instead of the installed packages. I guess I
misunderstood that part.

That's just sad. I woke up with the bright idea about removing the
packages from modulemd and then adding `sway-module-obsoletes` which
Obsoletes everything in the module that's behind f32-updates. We have
several packages like this since the last module rebuild.

If there's a cached non-updateable module metadata from fedora-modular
which is always considered by dnf together with the updated version from
fedora-updates-modular, that had no chance of working.

That's exactly how it works.
Unlike RPMs where usually the latest version is the result of a 
transaction, module metadata are additive.
Sum of all versions make a stream and it's content (incl. the old RPM 
versions) so you can eventually downgrade your package if you need to.
We know about the issue and we want to introduce some de-modularization 
feature, but we need to fix more serious issues first.





During the update to Fedora 33 a modular reset should happen anyway.
But even if it doesn't I believe that the packages would be upgraded
to non-modular, assuming their EVR is higher. I might be wrong thou,
because the concept of removing packages from modules and making them
non-modular is full of booby traps.


The module reset is essential, because otherwise DNF will keep using
fail-safe module data on disk and that would still keep the package
filtered.

I suggest making the change in F33 / Rawhide.
Rawhide shouldn't break, because there's only one (the latest) version
of each module in the repodata and if you release a new modulemd without
the package, everything should work as expected.

Your suggestion implies that there's a way to build different set of
packages for different releases within the same module stream, right?

Because with what Miro said, we are stuck with the requirement to update
every package we have in the module until f32 is EOL. I can't just drop
package x from modulemd and say "x won't be ever updated in f32 because
modularity" :(


The F33 change I suggested doesn't retrospectively apply on F32.
You're really stuck maintaining the package until F32 EOL.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Removing packages from module

[Daniel Mach]

Dne 17. 07. 20 v 11:04 Miro Hrončok napsal(a):

On 17. 07. 20 6:47, Aleksei Bavshin wrote:

In an optimistic case I'd like to believe that once the package is
removed from the modulemd file and a new module update is published, the
override magic hacked into dnf would stop masking package from
non-modular repository. NEVRA would be the only factor deciding the
update priority and when we push new build of the package into
f32-updates it would update the remaining modular build of the package
with lower EVR.


If the package was part of the module during the release of Fedora 32, 
dnf will see 2 versions/builds of the module-stream:


  - the one that existed during the release (containing the package)
  - the one from updates (no longer containing the package)

The "original" version will still filter out the non-modular version of 
the package from any transaction.


And given the way modular filtering works, a package cannot technically 
be "demodularized" this was during one release, unless you force a 
module reset from another package's scriptlet.


Thanks Miro,
That exactly describes what happens.


In a pessimistic case I expect that the package removed from the module
will never be updated by dnf in the same release cycle since the already
installed version originates from the modular repository. Upgrade to f33
would require module reset.


During the update to Fedora 33 a modular reset should happen anyway. But 
even if it doesn't I believe that the packages would be upgraded to 
non-modular, assuming their EVR is higher. I might be wrong thou, 
because the concept of removing packages from modules and making them 
non-modular is full of booby traps.


The module reset is essential, because otherwise DNF will keep using 
fail-safe module data on disk and that would still keep the package 
filtered.


I suggest making the change in F33 / Rawhide.
Rawhide shouldn't break, because there's only one (the latest) version 
of each module in the repodata and if you release a new modulemd without 
the package, everything should work as expected.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: RHEL 9 and modularity

[Daniel Mach]

Dne 24. 06. 20 v 11:56 Petr Pisar napsal(a):

On Wed, Jun 24, 2020 at 11:01:55AM +0200, clime wrote:

On Wed, 24 Jun 2020 at 10:35, Petr Pisar  wrote:


On Wed, Jun 24, 2020 at 10:22:38AM +0200, clime wrote:

On Wed, 24 Jun 2020 at 09:40, Petr Pisar  wrote:


On Wed, Jun 24, 2020 at 06:51:36AM +, Zbigniew Jędrzejewski-Szmek wrote:

Yes. Putting the "stream identification" into the package name is the
most natural solution, and has been floated various times.


This already happens. But not in Fedora. In RHEL, modular packages have
Modularitylabel RPM tag that carries the module name and stream.


Does "ModularityLabel" actually propagates to rpm package name or is
it just a "hidden" rpm attribute?


It's a RPM tag as well as a package name or a package version are the RPM
tags. I don't understand your question.


Well, the original sentence was: "Putting the "stream identification"
into the package *name* is the most natural solution".

And the answer was: "This already happens. But not in Fedora. In RHEL, ..."

But my question was answered, thank you.


I see. I focused on having the stream information on RPM level. Then the
answer is no, the package name does not contain the information.

My idea was that DNF could discriminate the same-name package using the
ModularityLabel tag instead of relying on modulemd documents delivered in the
repository metadata.


The "modularitylabel" is not going to help.
It's designed as a boolean flag.
If it holds any value, it indicates that the RPM is part of a module.
If it's empty, then the RPM is non-modular.
If you're looking for any sense of the header, it's probably closest to 
a reference to the module build it comes from.


RPMs are supposed to be re-used among modules/contexts and for that 
reason we cannot hardcode this relation directly to them.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: subtle issue with systemd, dnf 'greedy' obsoletes behaviour, and multiple repos

[Daniel Mach]

Dne 05. 06. 20 v 8:41 Kevin Kofler napsal(a):

The fact that Obsoletes from outdated packages are being considered is the
bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1748187
but the DNF developers refused to acknowledge this and just closed the bug
when the issue was worked around in the font package somehow.


This is nothing that can be fixed in DNF directly.
DNF passes packages to libsolv to resolve the transaction and it can 
only cherry-pick packages for the transaction or set transaction flags.


If you want change in behavior, please open a discussion with libsolv's 
upstream. The currently supported flags are: 
https://github.com/openSUSE/libsolv/blob/master/doc/libsolv-bindings.txt#constants


If I understand correctly how the sat-solver works, it's populated with 
a set of rules before the transaction is resolved and it's not possible 
to tweak in on fly as yum did. It could be possible to pre-process the 
packages and hide Obsoletes of older packages, but what if the latest 
package has broken deps and doesn't get into transaction? DNF in Fedora 
has best=0. In that case the older package would get into transaction 
and its Obsoletes would be ignored - and that's definitely something we 
don't want.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Modularity survey - results

[Daniel Mach]

Dne 19. 05. 20 v 16:56 Miro Hrončok napsal(a):

On 19. 05. 20 16:46, Christopher wrote:
Interesting that the survey shows that the most common response was 
that people use it "not at all" and the overall response was negative, 
but the reaction to that is, "improve the docs" and "works as 
intended". Am I the only one who thinks that the people pushing 
modularity aren't listening to the larger community?
FWIW I understand the action point for this survey as a way to improve 
the situation for environments where we are "stuck with modularity" 
(such as RHEL and EPEL 8) rather than a call for action that says "if we 
fix this, we'll proceed with Fedora modularization".



Exactly.



Our goal is to improve the *existing* implementation of Modularity (RHEL 
8 and EPEL 8 compatibility is important for us) and make it *available* 
to any projects under Fedora Project umbrella, which includes Fedora and 
EPEL 8. That's why we'll focus on the docs in the first place. The 
survey confirmed that people make a lot of assumptions about Modularity 
and interpret it in their own ways and this is source of many 
misunderstandings. When we're all on the same page, we can move on and 
work on individual fixes.


Our goal (I speak for the people who *currently* work on Modularity 
project at Red Hat) is *not* pushing anyone to use Modularity. It's up 
to Fesco, SIGs, spin maintainers and individual package maintainers to 
make their choices.


Please note that we are not the same people who created Modularity, we 
took it over recently and we are expected to maintain it, not to 
redesign it from scratch. Please give us a chance to prove that we 
listen to the community.





I interpret the survey results that some people in Fedora would use 
modules to build and distribute multiple versions of their software 
stacks, but the increased complexity and related issues are not worth it.


I believe that to make Modularity successful and useful for the 
community, we need to simplify it and better integrate it with the rpm 
ecosystem. Make it more straightforward for those who build or 
distribute modules and nearly invisible for those who consume them.


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Modularity survey - results

[Daniel Mach]

Hello everyone,
We have finally evaluated all of your responses to the Modularity 
survey. You can find the results posted on the Fedora community blog[1].


Thanks to all of you who filled the survey and provided detailed 
explanation of what works and what not.



[1] https://communityblog.fedoraproject.org/modularity-survey-results/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: RFC: Feature macros (aka USE flags)

[Daniel Mach]

Dne 27. 04. 20 v 19:00 Stephen Gallagher napsal(a):

On Mon, Apr 27, 2020 at 10:47 AM Daniel Mach  wrote:


I'm wondering if this is your personal initiative or if you're sync with
ELN people. I emailed them in January about the very same idea (and I
used the very same name; we both seem to like Gentoo), we exchanged
couple emails, but never got an answer if this is the way to go. Since I
have a lot of problems of my own (dnf, rpm, modularity), I did not want
to start this as my personal initiative.



Yes, Petr is in fact ELN people and he's working on this at least
partly within that context (though the benefits are not exclusive to
ELN).

We looked at your approach, but it seemed like you were advocating for
dealing with this somehow at the libdnf layer, which we didn't think
was the right place. If we misunderstood, that's on us.
Most likely. It had nothing to do with libdnf, it was a generic concept 
based on centralized macros defined in the build roots.





I'm really glad that someone's looking into this finally.
BTW, the new libdnf spec is using this approach already:
https://github.com/rpm-software-management/libdnf/blob/dnf-5-devel/libdnf.spec



Well, not exactly. But it's similar. The main advantage here is that
we can define some things globally for all packages.a

Yes, that was exactly the point in my proposal.
Building for another use case would mean just setting build root macros 
and building the package differently, no spec change needed.





Since I'm part of RPM team too, I hope they won't mind if I'll speak for
them :) Don't you rather want to work with us on extending the existing
with/without macros? I'd prefer to improve the existing approach over
creating something brand new. We could also reuse existing rpmbuild
--with/--without arguments and ideally remain backwards compatible.


Could you explain a bit more what this means to you? I'm not sure what
you would want to do in RPM itself.
rpmbuild supports --with and --without options for setting the %with 
macros already. That allows you to make a local build with custom %with 
values. I suppose you'll need something similar for the new feature. 
From my perspective it would be better to re-use the existing options 
rather than introducing --use/--dont-use or something similar.



___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: RFC: Feature macros (aka USE flags)

[Daniel Mach]

I'm wondering if this is your personal initiative or if you're sync with 
ELN people. I emailed them in January about the very same idea (and I 
used the very same name; we both seem to like Gentoo), we exchanged 
couple emails, but never got an answer if this is the way to go. Since I 
have a lot of problems of my own (dnf, rpm, modularity), I did not want 
to start this as my personal initiative.


I'm really glad that someone's looking into this finally.
BTW, the new libdnf spec is using this approach already: 
https://github.com/rpm-software-management/libdnf/blob/dnf-5-devel/libdnf.spec


Since I'm part of RPM team too, I hope they won't mind if I'll speak for 
them :) Don't you rather want to work with us on extending the existing 
with/without macros? I'd prefer to improve the existing approach over 
creating something brand new. We could also reuse existing rpmbuild 
--with/--without arguments and ideally remain backwards compatible.


Besides this, +1 from me


Dne 27. 04. 20 v 13:19 Petr Šabata napsal(a):

Based on the recent discussions around %fedora/%rhel macros and ELN,
and %bcond generally being confusing to work with, I came up with a
distribution-wide feature that defines generic feature keywords and
associated helper macros that packages can check in build-time
conditionals.

The key advantage here is the defaults are defined by the buildroot,
not the package. The package is just a building block.

I'd like some input to improve this and unless this turns out to be a
really bad idea, I intend to submit it as a change proposal. Even
though the more packages use it the more beneficial it gets, it's, of
course, perfectly optional.

Details in the gist:
https://gist.github.com/contyk/0f0585c57976ca18a293b3566408

Cheers,
P

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: RFC: Feature macros (aka USE flags)

[Daniel Mach]

I'm wondering if this is your personal initiative or if you're sync with 
ELN people. I emailed them in January about the very same idea (and I 
used the very same name; we both seem to like Gentoo), we exchanged 
couple emails, but never got an answer if this is the way to go. Since I 
have a lot of problems of my own (dnf, rpm, modularity), I did not want 
to start this as my personal initiative.


I'm really glad that someone's looking into this finally.
BTW, the new libdnf spec is using this approach already: 
https://github.com/rpm-software-management/libdnf/blob/dnf-5-devel/libdnf.spec


Since I'm part of RPM team too, I hope they won't mind if I'll speak for 
them :) Don't you rather want to work with us on extending the existing 
with/without macros? I'd prefer to improve the existing approach over 
creating something brand new. We could also reuse existing rpmbuild 
--with/--without arguments and ideally remain backwards compatible.


Besides this, +1 from me


Dne 27. 04. 20 v 13:19 Petr Šabata napsal(a):

Based on the recent discussions around %fedora/%rhel macros and ELN,
and %bcond generally being confusing to work with, I came up with a
distribution-wide feature that defines generic feature keywords and
associated helper macros that packages can check in build-time
conditionals.

The key advantage here is the defaults are defined by the buildroot,
not the package. The package is just a building block.

I'd like some input to improve this and unless this turns out to be a
really bad idea, I intend to submit it as a change proposal. Even
though the more packages use it the more beneficial it gets, it's, of
course, perfectly optional.

Details in the gist:
https://gist.github.com/contyk/0f0585c57976ca18a293b3566408

Cheers,
P

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Modularity Survey

[Daniel Mach]

Updated.
Thanks both of you for the suggestion.


Dne 08. 04. 20 v 19:59 Adam Williamson napsal(a):

On Wed, 2020-04-08 at 13:47 -0400, Alex Scheel wrote:

Hey Daniel, do you mind updating the GDPR compliance tag to include
Google?


Right, this is all I intended in the first place :) A simple:

-The raw data will not be provided to anyone else at Red Hat or any 3rd parties
+The raw data will not be provided to anyone else at Red Hat or any 3rd parties 
(except Google)

would do the trick just fine. We didn't need a big thread about it...


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Modularity Survey

[Daniel Mach]

Hello everyone,

On behalf of Red Hat's Modularity team, I'd like to ask you to fill out 
a survey on Modularity:

https://docs.google.com/forms/d/e/1FAIpQLScOA97rGONieSOYmlZLsHdkq-EhdePZ4IN3RwOjJKd1F1a9sw/viewform?usp=sf_link

Our goal is to use your feedback to improve Modularity, its 
documentation and hopefully fix any issues you may have.



Modularity Survey
-
The purpose of this survey is to get feedback on Modularity.

It is divided into 4 sections:
* Information about yourself (optional)
* Modularity & you
* Problems with Modularity you may have experienced
* Glossary review - what do you think the terms mean

Privacy / GDPR:
* The raw data incl. any personal information you provide will be shared 
only with Red Hat's Modularity team (approx. 10 people) to evaluate the 
survey
* The raw data will not be provided to anyone else at Red Hat or any 3rd 
parties

* Aggregated (anonymous) results of the survey will be published

Thank you for your cooperation.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Introduce module Obsoletes and EOL

[Daniel Mach]

Dne 31. 03. 20 v 10:14 Zbigniew Jędrzejewski-Szmek napsal(a):

On Tue, Mar 31, 2020 at 09:25:45AM +0200, Vít Ondruch wrote:

Wouldn't be easier to use something we already have? E.g.
https://src.fedoraproject.org/rpms/fedora-obsolete-packages


Obsoletes in individual rpms have no effect on modules, and any
enabled module "shadows" non-modular rpms that the module declared,
i.e. makes dnf consider only rpms from the module, completely ignoring
rpms from the normal repo, no matter what EVRA they have.
If that sounds wrong to you, you are not alone. But things are as
they are, and this Change is about making upgrades work within the
design of Modularity, hopefully avoiding stuff like [1,2,3,4,5].


I don't think it's going to fix most of them, but it's a move towards a 
saner system where modules support packaging best practices (adding 
Obsoletes this time).




[1] 
https://fedoraproject.org/wiki/Common_F31_bugs#Eclipse_module_has_been_reset_to_avoid_shadowing_non-modular_rpms
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1780827

This change won't help with [1] and [2].
I'm quite happy with the workaround we used, because this was a one-time 
fix for a quite uncommon situation.



[3] https://bugzilla.redhat.com/show_bug.cgi?id=1747408

IIRC the libgit2 problem was caused by change in stream dependencies.
This will require an additional change in how Context is used.
The overall idea is to introduce upgrade paths among Contexts by simply 
stopping changing them. If N:S:C represented a virtual repository 
(potentialy also a branch and a build target), we'd finally have upgrade 
paths in modularity. DNF has to guess now and no wonder it errors out 
from time to time. But this change will require more time, as it changes 
stream expansion and the whole build process. I don't think we'll be 
able to make it in the Fedora 33 time frame.



[4] https://bugzilla.redhat.com/show_bug.cgi?id=1807832

This is a modularity feature.
Modules can provide older packages than distro provides.
That's why it applies the RPM filtering - to hide newer RPMs provided by 
distro.

This must be solved by having a good packaging policy and tests.


[5] https://bugzilla.redhat.com/show_bug.cgi?id=1806303

The problem was in misunderstanding how defaults should be removed.
Modularity works across repos and 'updates' don't automatically override 
'fedora' -> empty defaults with higher priority must be published in 
'updates'




Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Introduce module Obsoletes and EOL

[Daniel Mach]

Dne 31. 03. 20 v 10:00 Petr Pisar napsal(a):

On Tue, Mar 31, 2020 at 09:25:45AM +0200, Vít Ondruch wrote:

Wouldn't be easier to use something we already have? E.g.
https://src.fedoraproject.org/rpms/fedora-obsolete-packages


DNF implements modules as an layer above RPM packages. Thus you cannot control
modules metadata from an RPM metadata.

But as a delivery mechanism for the new modular metadata it could work. But
only for those who has already installed the fedora-obsolete-packages package.
And not for a fresh new installation or an upgrade from an non-up-to-date
system. Imagine you have an EOLed Fedora 29 and you find out you need to 
obsolete
a module in Fedora 30. An impossible task because Fedora 29 does no longer
receive any updates. That's the same problem why module defaults are delivered
on YUM repository level, not inside an RPM package.


Exactly.
Storing the metadata in an RPM would mean you can use it *after* you 
finish a transaction.
It would also cause problems to Anaconda, Mock and any other tools that 
work with clean installs and should have the up-to-date information from 
the beginning.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Introduce module Obsoletes and EOL

[Daniel Mach]

Dne 30. 03. 20 v 19:20 Zbigniew Jędrzejewski-Szmek napsal(a):

On Mon, Mar 30, 2020 at 10:30:51AM -0400, Ben Cotton wrote:


document: TBD
version: 1
data:
   # A string representing UTC date in ISO 8601 format: -MM-DD[T ]HH:MMZ
   # When merging, entries with a newer 'modified' value will override
any earlier values.
   # MANDATORY
   modified: 2020-03-19T23:26Z

   # A boolean option to cancel/reset all previously specified obsoletes
   # Example: repo 'fedora' has Obsoletes:nodejs:12; we want to bring
nodejs:12 back in 'updates'
   # If used, following options will be ignored: eol_date, obsoleted_by
   # OPTIONAL
   : 


What obsoletes would be covered by this? Those for this particular stream?

Obsoletes for specified Name, Stream and Context(s) (if present).




   # A string representing a Name of a module that is EOLed
   # MANDATORY
   module: nodejs

   # A string representing a Stream of a module that is EOLed
   # MANDATORY
   stream: 11

   # A string representing a Context of a module that is EOLed
   # If not specified, all contexts get EOLed.
   # NOTE: consider specifying a list of contexts
   # OPTIONAL
   context: aabbccddee

   # A string representing UTC date in ISO 8601 format: -MM-DD[T ]HH:MMZ
   # It is strongly recommended to keep HH:MM to 00:00.
   # If not specified, the module is EOLed immediately.
   OPTIONAL
   eol_date: 2020-03-19T00:00Z


Normally we want the obsoletes to happen when doing a 'dnf system-upgrade'
operation, and not based on time. Does this allow us to tell dnf for this
happen during an upgrade, but not before?
There are scenarios when you don't run system-upgrade and you still want 
the obsoletes. Consider Rawhide. We'll need a way how to trigger modular 
obsoletes even during normal upgrade/distro-sync transactions.


Setting eol_date to the future allows informing a user about upcoming 
obsoleting event so they can eventually migrate manually.





   # A string describing the change, reason, etc.
   # MANDATORY
   message: "Module stream nodejs:11 is no longer supported. It is
recommended to switch to nodejs:12"

   # If a stream is not EOLed but Obsoleted, provide details about the
obsoleting stream:
   # OPTIONAL
   obsoleted_by:
 module: nodejs
 stream: 12


Is the obsoleting module enabled automatically?

Kind of; I expect following behavior:
* print a warning by default and have a possibility to manually turn 
stream obsoletes on (must have on Rawhide)

* an automated stream switch during system-upgrade




== User Experience ==
Fedora users experience upgradability problems when upgrading to Fedora 32
when they have any module streams enabled.

If a stream no longer exists on the version of Fedora they are upgrading to,
DNF used to error out on resolving modular dependencies which was a
clear release blocker.
To workaround this case, all module streams are reset during the system upgrade.
By doing this, modularity users lose information about enabled streams
and they need to re-enable them by hand.

This Change aims at removing the upgradability problems and allowing
Fedora modularity users
to upgrade their systems while keeping their streams enabled, reset or
switched (obsoleted)
according to newly provided modular metadata.


Thanks, that sounds like a reasonable UX.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Introduce module Obsoletes and EOL

[Daniel Mach]

Dne 30. 03. 20 v 17:10 Petr Pisar napsal(a):

On Mon, Mar 30, 2020 at 10:30:51AM -0400, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/Module_Obsoletes_and_EOL

[...]

*** When a module stream gets removed from a Fedora release, the
maintainer of the module stream must provide a modulemd document with
Obsoletes or EOL data.


I'd like to point out that Fedora Project has already possed EOL dates. They
are provided by packagers when requesting for new dist-git branch (i.e. when
creating a new stream) and they are stored in Fedora PDC.

Examples for perl:5.26
:

 {
 "id": 691015,
 "sla": "bug_fixes",
 "branch": {
 "id": 345929,
 "name": "5.26",
 "global_component": "perl",
 "type": "module",
 "critical_path": false,
 "active": false
 },
 "eol": "2019-12-01"
 }


I'll take a look.
This seems to be a reasonable input for checking which streams should be 
EOLed. Maybe we could automate creating the modulemd obsoletes metadata 
based on this.



So if you execute "dnf module list perl" in Fedora 30 you should not see
perl:5.26 accoding to this change.

Provided Fedora Project is going to keep requesting the EOL datum on creating
a branch, pungi could query PDC and automatically generate the
module-obsolete documents for streams and contexts that require those streams.

Sounds reasonable.



The only missing piece of information is the obsoleting stream. Either the
maintainer could edit the PDC data with this information when it is known (as
he already can edit the eol date by a rel-eng ticket
; see module_eol template type), or the
maintainer could supply its own module-obsolete document (how?) that would
inhibit the generating the document for the same stream by pungi.
To be honest, I don't know how it works exactly and how to tweak the 
existing process, but I believe that people from Fedora infra will help 
me with that.




In both cases it would be great if an Fedora infrastrucure send a notification
to the maintainer the OEL is approaching and he should provide the data (e.g.
by filing a bug report into Bugzilla against that module).

-- Petr


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Announcing start of DNF 5 development

[Daniel Mach]

Dne 12. 03. 20 v 19:26 Pat Riehecky napsal(a):

I realize I'm thinking about the Pie in the Sky, but:

Would it be possible for 'microdnf' to become the base for 'dnf' so
that extra 'dnf' functionality is added via some kind of
modules/plugins/etc behaviour?  Perhaps "somehow" it could (for
example) find out "oh I've got python, lets load those feature bits
too".

This could help keep them fully compatible and let folks using dnf
based installers look at pulling in just the features they require.



I don't think we're ready for this yet. We need to work on improving 
libdnf and delivering a dbus interface in the first place as announced. 
Once we're finished with these changes, we'll definitely look into 
making python optional.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Announcing start of DNF 5 development

[Daniel Mach]

Dne 12. 03. 20 v 6:21 Chris Murphy napsal(a):

On Thu, Mar 5, 2020 at 8:30 AM Daniel Mach  wrote:




Dne 04. 03. 20 v 23:01 Neal Gompa napsal(a):

On Wed, Mar 4, 2020 at 4:37 PM Zbigniew Jędrzejewski-Szmek
 wrote:

Are you going to use sd-bus for the dbus library?



I'd hope not, given that we have cross-distro usage of DNF now, and a
couple of them don't have systemd.


Do you know which distros do not have systemd?

We have evaluated sd-bus to be the best dbus client available, but we
may have underestimated it's adoption.

Couldn't systemd team split it into a separate library independent on
the rest of the systemd eco-system? :)


Is varlink applicable here? I mention it because I've come across it
being used by systemd-homed. It's intended to be available during
early boot.



We have considered both dbus and varlink and decided to use dbus. Dbus 
has better adoption and we'll be replacing a dbus service (PackageKit) 
with another.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Announcing start of DNF 5 development

[Daniel Mach]

Dne 05. 03. 20 v 19:38 J. Randall Owens napsal(a):

On 04/03/2020 18:03, Daniel Mach wrote:

Hello everyone,
I'm pleased to announce start of DNF 5 development. We are planning to
deliver a module stream or a COPR repo during Fedora 33 development for
early adopters and tool developers and we're hoping in getting a stable
version into Fedora 34.


More details follow.


We've managed to drop a lot of redundant code across the whole DNF stack
in the past years, but we have reached a point when it's nearly
impossible to consolidate the code any further without breaking the
API/ABI. Especially with PackageKit being dead[1], we can't move with
the old "libhif" API in libdnf, because making any bigger changes to
PackageKit is clearly out of scope.

[1]
https://blogs.gnome.org/hughsie/2019/02/14/packagekit-is-dead-long-live-well-something-else/



That's why we decided to start working on a new version of the DNF
stack: DNF 5. And this is the plan:


Priorities
--
1. Consistency, documentation and user experience is the top priority.
2. Compatibility on the command line level.
3. Compatibility on the API level.



In favour of 1., but going against 2., I've wanted since the YUM days to
see more consistency in the compound-word* options. E.g., --enablerepo
but --skip-broken, and I have a hard time remembering which ones get the
hyphen inbetween*, and which ones don't. Could we settle on one way or
the other, right now quite the majority are without hyphen, and
deprecate the other (but presumably enable both spellings for a good
long while, perhaps with a warning if someone spells it the old way)?
Probably not take away the old spellings as soon as DNF 5, but it might
be a good time to add the more consistent spellings.

* Yes, I did that on purpose.



I'll definitely put this on the list.

We're primarily aiming at even worse stuff:
$ dnf list bash -> displays bash package
$ dnf list available -> displays all available packages
$ dnf list installed -> displays all installed packages
$ dnf list all -> displays all packages
What if there were packages named 'all', 'available' or 'installed'?
These are yum compatibility quirks which should be replaced with proper 
--options. There's definitely much more than this example.


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Announcing start of DNF 5 development

[Daniel Mach]

Dne 05. 03. 20 v 13:02 Marcin Juszkiewicz napsal(a):

W dniu 04.03.2020 o 19:03, Daniel Mach pisze:

Hello everyone, I'm pleased to announce start of DNF 5 development.



microdnf

  > Microdnf is becoming important because it's part of

many containers due to its small footprint.


[root@puchatek hrw]# ldd /bin/microdnf |wc -l
70Following deps should be going away:

libglib-2.0.so.0
libgobject-2.0.so.0
libffi.so.6
libpeas-1.0.so.0
libgmodule-2.0.so.0
libgirepository-1.0.so.1
libgio-2.0.so.0



hrw@j13-qrep-04:ceph-12.2.11$ ldd /usr/bin/apt|wc -l
20

Are there plans for picodnf then? Or cutting amount of
libraries used by microdnf?


I don't think we can drop more without losing functionality.
If you take a closer look, libdnf links to libcurl which pulls most of 
the dependencies.




My problem with DNF is Python. There is a huge amount of
packages which need to be in proper state to be able to use
dnf. I remember when I did some experiments with RHEL7 and
managed to get to the point where 'yum' was unable to help
as Python was broken.


This probably makes you one of the users we're improving microdnf for.
I hope that someday we'll get microdnf close to 100% feature parity with 
DNF (Python plugins excluding obviously). But that's going to take a 
while...

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Announcing start of DNF 5 development

[Daniel Mach]

Dne 04. 03. 20 v 23:01 Neal Gompa napsal(a):

On Wed, Mar 4, 2020 at 4:37 PM Zbigniew Jędrzejewski-Szmek
 wrote:

Are you going to use sd-bus for the dbus library?



I'd hope not, given that we have cross-distro usage of DNF now, and a
couple of them don't have systemd.


Do you know which distros do not have systemd?

We have evaluated sd-bus to be the best dbus client available, but we 
may have underestimated it's adoption.


Couldn't systemd team split it into a separate library independent on 
the rest of the systemd eco-system? :)

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Announcing start of DNF 5 development

[Daniel Mach]

Dne 04. 03. 20 v 22:34 Zbigniew Jędrzejewski-Szmek napsal(a):

On Wed, Mar 04, 2020 at 07:03:01PM +0100, Daniel Mach wrote:

Hello everyone,
I'm pleased to announce start of DNF 5 development. We are planning
to deliver a module stream or a COPR repo during Fedora 33
development for early adopters and tool developers and we're hoping
in getting a stable version into Fedora 34.


More details follow.


We've managed to drop a lot of redundant code across the whole DNF
stack in the past years, but we have reached a point when it's
nearly impossible to consolidate the code any further without
breaking the API/ABI. Especially with PackageKit being dead[1], we
can't move with the old "libhif" API in libdnf, because making any
bigger changes to PackageKit is clearly out of scope.

[1] 
https://blogs.gnome.org/hughsie/2019/02/14/packagekit-is-dead-long-live-well-something-else/


That's why we decided to start working on a new version of the DNF
stack: DNF 5. And this is the plan:


Priorities
--
1. Consistency, documentation and user experience is the top priority.
2. Compatibility on the command line level.
3. Compatibility on the API level.


Maintenance
---
The existing DNF 4 stack stays in the current Fedoras and Red Hat
Enterprise Linux 8. We'll keep maintaining it in dnf-4-master
branches on GitHub. PackageKit and rpm-ostree will stay on libdnf
from the DNF 4 stack.


The existing Python API in DNF
--
The Python API in DNF stays. We'll do our best to keep it working.
If there is an incompatible change, we'll communicate and document
it properly.


The new API in libdnf
-
All business logic will move from DNF (Python) to libdnf (C++). This
is the only way to ensure that package managers work identically
across the whole distribution. We'll start with C++ API and
auto-generated Python bindings via SWIG. We'll focus on the Python
bindings which are required by DNF and we will do our best to
provide bindings for Go, Perl5 and Ruby as well. C API will be
created later when the C++ API is stable. At that moment rpm-ostree
will be ported to the new C API.


hawkey
--
Hawkey Python API is going away and will be replaced with libdnf Python API.


DNF
---
DNF stays as the primary command-line package manager. The overall
functionality remains. We don't anticipate any negative impact of
the API rewrite on the end-users. We have built an extensive test
suite (over 1400 scenarios) that will help us to ensure that. The
argument parser and outputs may slightly change in some cases to
provide a more consistent user-experience. All such cases will be
properly documented.


microdnf

Microdnf is becoming important because it's part of many containers
due to its small footprint. We're getting feedback that users would
appreciate something closer to DNF. We'll focus on implementing a
subset of DNF's functionality and improving the user experience.
100% feature parity with DNF is currently out of scope.



Hi,

the roadmap is ambitious, but not impossible. Good luck!

Thanks!




Roadmap (tentative)
---
* Mar 2020 - making the bigger API changes, upstream code barely compiles
* May 2020 - COPR repo with first development snapshots
* Jun 2020 - F33 module available for early adopters and tool developers
* Oct 2020 - DNF 5 landing in F34 Rawhide
* Feb 2021 - DNF 5 replacing DNF 4 in stable Fedora



DBus service

DNF team has decided to create a new DBus service replacing
PackageKit to provide an interface to GUI applications. It's
probably going to take a while because we're planning to start from
scratch.


Do you plan to make normal 'dnf' calls go through the dbus api?
(And e.g. provide a single cache and privilege escalation through
packagekit)?

No, dnf stays as it is, it's not going to switch to dbus.
Single cache is part of the plan.



Apart from the dbus api, do you plan to provide some graphical
application that uses this api?

Definitely not at this moment.
We need to build the new service and integrate it with the existing 
tools first. We may create a simple command-line tool replacing pkcon, 
something like microdnf over dbus.




Are you going to use sd-bus for the dbus library?


Yes, it's going to be sd-bus + sdbus-cpp:
* https://github.com/Kistler-Group/sdbus-cpp
* https://src.fedoraproject.org/rpms/sdbus-cpp
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Announcing start of DNF 5 development

[Daniel Mach]

Hello everyone,
I'm pleased to announce start of DNF 5 development. We are planning to 
deliver a module stream or a COPR repo during Fedora 33 development for 
early adopters and tool developers and we're hoping in getting a stable 
version into Fedora 34.



More details follow.


We've managed to drop a lot of redundant code across the whole DNF stack 
in the past years, but we have reached a point when it's nearly 
impossible to consolidate the code any further without breaking the 
API/ABI. Especially with PackageKit being dead[1], we can't move with 
the old "libhif" API in libdnf, because making any bigger changes to 
PackageKit is clearly out of scope.


[1] 
https://blogs.gnome.org/hughsie/2019/02/14/packagekit-is-dead-long-live-well-something-else/



That's why we decided to start working on a new version of the DNF 
stack: DNF 5. And this is the plan:



Priorities
--
1. Consistency, documentation and user experience is the top priority.
2. Compatibility on the command line level.
3. Compatibility on the API level.


Maintenance
---
The existing DNF 4 stack stays in the current Fedoras and Red Hat 
Enterprise Linux 8. We'll keep maintaining it in dnf-4-master branches 
on GitHub. PackageKit and rpm-ostree will stay on libdnf from the DNF 4 
stack.



The existing Python API in DNF
--
The Python API in DNF stays. We'll do our best to keep it working. If 
there is an incompatible change, we'll communicate and document it properly.



The new API in libdnf
-
All business logic will move from DNF (Python) to libdnf (C++). This is 
the only way to ensure that package managers work identically across the 
whole distribution. We'll start with C++ API and auto-generated Python 
bindings via SWIG. We'll focus on the Python bindings which are required 
by DNF and we will do our best to provide bindings for Go, Perl5 and 
Ruby as well. C API will be created later when the C++ API is stable. At 
that moment rpm-ostree will be ported to the new C API.



hawkey
--
Hawkey Python API is going away and will be replaced with libdnf Python API.


DNF
---
DNF stays as the primary command-line package manager. The overall 
functionality remains. We don't anticipate any negative impact of the 
API rewrite on the end-users. We have built an extensive test suite 
(over 1400 scenarios) that will help us to ensure that. The argument 
parser and outputs may slightly change in some cases to provide a more 
consistent user-experience. All such cases will be properly documented.



microdnf

Microdnf is becoming important because it's part of many containers
due to its small footprint. We're getting feedback that users would
appreciate something closer to DNF. We'll focus on implementing a subset 
of DNF's functionality and improving the user experience. 100% feature 
parity with DNF is currently out of scope.



DBus service

DNF team has decided to create a new DBus service replacing PackageKit 
to provide an interface to GUI applications. It's probably going to take 
a while because we're planning to start from scratch.



Roadmap (tentative)
---
* Mar 2020 - making the bigger API changes, upstream code barely compiles
* May 2020 - COPR repo with first development snapshots
* Jun 2020 - F33 module available for early adopters and tool developers
* Oct 2020 - DNF 5 landing in F34 Rawhide
* Feb 2021 - DNF 5 replacing DNF 4 in stable Fedora
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Points of contact for Modularity

[Daniel Mach]

Petr,
thank you for the introduction and for your and your team's work on 
Modularity for last several years.


DNF and Modularity are under one roof now. We believe we have sufficient 
knowledge to continue the work you have started and that we'll put fresh 
energy in the project and hopefully take it to the next level.



What's next?
In the first place we must focus on resolving the existing Fedora issues 
such as: Bug 1767351 - Cannot upgrade to Fedora 32: Modules blocking the 
upgrade path


Then we will focus on overall user experience - that includes both 
end-users and packagers.


We're currently creating long-term plans and when the new Modularity 
Objective is drafted, we'll ask in this mailing list for feedback.


If you can give any feedback already (Modularity in general, 
documentation, user experience, DNF misbehaving or anything else you 
find important), please file a bug on dnf (we'll make sure it gets to 
the right hands) or send me an email.


thanks,
Daniel


Dne 19. 02. 20 v 17:07 Petr Šabata napsal(a):

Just a minor FYI announcement:

As most of the original Modularity Team is moving on to face new
challenges, Red Hat's effort will now be primarily represented by the
software management group, with Daniel Mach coming up with an updated
Modularity Objective in the following weeks.

In essence, everything remains the same; continue using the Pagure
tracker and our IRC channel when interacting with the team -- just
expect to see "new" faces from now on. The weekly meetings are
temporarily cancelled until the group agrees on a new time slot and
way of working.

Thank you,
P
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: dnf exit code

[Daniel Mach]

There seem to be 2 related problems:
1) Only the image on docker.io doesn't work. If you use 
registry.fedoraproject.org/fedora:rawhide, it works as expected.


2) Fedora infra is unreliable. When the image from 
registry.fedoraproject.org cannot be downloaded, there's a fallback to 
docker.io which contains a different image. Running the command several 
times in a row produces "random" results:



$ podman pull fedora:rawhide
Trying to pull registry.fedoraproject.org/fedora:rawhide...
...
a3689ffc0440236cbe0530e9ead8b76d0f601a33cd5ce08409d2d96d70abf56d


$ podman pull fedora:rawhide
Trying to pull registry.fedoraproject.org/fedora:rawhide...
...
a3689ffc0440236cbe0530e9ead8b76d0f601a33cd5ce08409d2d96d70abf56d


$ podman pull fedora:rawhide
Trying to pull registry.fedoraproject.org/fedora:rawhide...
  received unexpected HTTP status: 503 Service Temporarily Unavailable
Trying to pull registry.access.redhat.com/fedora:rawhide...
  name unknown: Repo not found
Trying to pull registry.centos.org/fedora:rawhide...
  manifest unknown: manifest unknown
Trying to pull docker.io/library/fedora:rawhide...
...
e13031c001a8b4a574e3088e2d1ab331d72d821804ccacdd41bf5662ae02cc98



Dne 12. 02. 20 v 0:37 Christoph Junghans napsal(a):

Hi,

I am getting an exit code 143 when running dnf in docker, example:
$ cat Dockerfile
FROM fedora:rawhide
RUN dnf install -y openmpi-devel
$ docker build .
...
Running scriptlet: systemd-245~rc1-2.fc32.x86_64  121/121
Running scriptlet: systemd-udev-245~rc1-2.fc32.x86_64
121/121The command '/bin/sh -c dnf install -y openmpi-devel' returned
a non-zero code: 143

Only on rawhide, f31 works fine, i.e.
$ cat Dockerfile
FROM fedora:31
RUN dnf install -y openmpi-devel
$ docker build .
Complete!
Removing intermediate container 85bb626aff58
  ---> e9a30f6992d7
Successfully built e9a30f6992d7

Any ideas?

Thanks,

Christoph



___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: kernel rpm split

[Daniel Mach]

Dne 07. 02. 20 v 13:47 Neal Gompa napsal(a):

On Fri, Feb 7, 2020 at 4:00 AM Daniel Mach  wrote >> Dne 06. 
02. 20 v 19:54 Zbigniew Jędrzejewski-Szmek napsal(a):

- move the Requires:linux-firmware (or change to Recommends) from kernel-core,
have kernel Requires:linux-firmware

It would be nice to introduce hardware specific Requires implemented.
If you have the hardware, a Provide would be dynamically generated (an
udev rule?) and used in rich dependencies to pull the firmware package
you need.

I believe SUSE is doing something similar already, but I don't quite
like references to PCI-IDs, for example:
Supplements: (kernel-default and
namespace:modalias(pci:v80EEdBEEFsv*sd*bc*sc*i*))




We already have kmods generate modalias provides if they're associated
with hardware, I think? So it wouldn't be that bad to have DNF trigger
inject virtual supplements that would pull in kernel modules.



I'm not opposed to that as long as someone outside the DNF team manages 
the rules. We are in the package management business and know only a 
little about hardware, firmware, kmods and how they relate to each other.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: kernel rpm split

[Daniel Mach]

Dne 06. 02. 20 v 19:54 Zbigniew Jędrzejewski-Szmek napsal(a):

Hello kernel maintainers, hello Fedora developers,

I'm looking into the split of kernel packages. The split into subpackages
seems interesting, but there are many dependencies between the packages,
so it is usual to end up with all of them installed.

E.g. for a simple VM, by design, kernel-core contains a basic set of
modules that should be enough to boot. I see that in a standard
libvirt guest, the only modules from kernel-modules that are loaded
are for sound hardware and "usbnet", all which I'd be fine without.

Another example: we'd like to explore building an initramfs directly
from rpms, without dracut, only systemd and a standard packages to
bring up the hardware. Some modules need to be installed, so the
kernel can load the from the initramfs, but the kernel itself should
not, since it is provided "externally" by the boot loader.

But:
the basic modules are in one rpm with kernel-core
kernel-core Requires linux-firmware (which is 240MB)
kernel-modules Requires kernel-uname-r, which is provided by kernel-core
kernel Requires kernel-core-uname-r, kernel-modules-uname-r

Would it be possible to make some changes:

- split out the modules from kernel-core package into a new subpackage
   kernel-basic-modules, kernel-core can Recommend or Require it
If you're after generating a minimal initramfs, this still doesn't solve 
the problem. In many cases you need kernel-core *and* several modules.


An option would be splitting the modules into individual RPMs and 
installing only those we need. But it is a bad idea (remember texlive 
and it's subpackage explosion).


I was thinking if you couldn't look into using a similar approach to the 
%_install_langs macro - simply define modules you want installed, the 
others would be skipped.




- remove the Requires on kernel-core (or change to Recommends) from
   kernel-modules, so it can be installed standalone

- move the Requires:linux-firmware (or change to Recommends) from kernel-core,
   have kernel Requires:linux-firmware

It would be nice to introduce hardware specific Requires implemented.
If you have the hardware, a Provide would be dynamically generated (an 
udev rule?) and used in rich dependencies to pull the firmware package 
you need.


I believe SUSE is doing something similar already, but I don't quite 
like references to PCI-IDs, for example:
Supplements: (kernel-default and 
namespace:modalias(pci:v80EEdBEEFsv*sd*bc*sc*i*))





I think this would be useful for playing with various minimization scenarios.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Modularity and the system-upgrade path

[Daniel Mach]

On 10/7/19 8:55 PM, Simo Sorce wrote:

I have to ask,
given containers are so popular and can deal with any dependency
without conflicting with system installed binaries, should we really
continue with this very complicated modular design ?


There are only few people who fully understand how modularity works 
today (contyk who designed modulemd, jmracek and me who implemented the 
DNF part and few others). I agree that the modular design should be 
simplified. If we don't lower the bar, the complexity might prevent from 
wider adoption.


As a former release engineer, I'm personally unhappy about lack of 
upgrade paths between module contexts and I believe that fixing this 
part of modularity design could lead to desired simplification. 
Unfortunately based on discussion I had with contyk yesterday, I don't 
believe it's achievable without making *huge* changes in the modularity 
design and the build infrastructure/process.




Shouldn't we go back to have default packages and then defer to
"containers" for applications (and their dependencies) that need to
deviate from system defaults for any reason ?


I don't think containers can replace modularity. They need to coexist. 
If we want to create containers built on top of a distribution (no 
randomly picked bits from the internet, reproducible builds, security, 
...), we need a way to distribute multiple versions of the software 
(module streams) and they frequently need to be built against each other 
 (module contexts).





Simo.

On Fri, 2019-10-04 at 10:57 -0400, Stephen Gallagher wrote:

Right now, there are two conflicting requirements in Fedora Modularity
that we need to resolve.

1. Once a user has selected a stream, updates should follow that
stream and not introduce incompatiblities. Selected streams should not
be changed without direct action from the user.
2. So far as possible, Modularity should be invisible to those who
don't specifically need it. This means being able to set default
streams so that `yum install package` works for module-provided
content.

Where this becomes an issue is at system-upgrade time (moving from
Fedora 30->31 or continuously tracking Rawhide). Because of
requirement 1, we cannot automatically move users between streams, but
in the case of release upgrades we often want to move to a new default
for the distribution.


The Modularity WG has generally agreed that we want and need to
support behavior of the following use-cases:


Use Case 1:

On Fedora 30, user Alice runs

yum install Foo

The package "Foo" is provided by a module "foo" with a default stream
"v1.0". Because it's available in a default stream, the package is
installed and the module stream "foo:v1.0" is implicitly enabled for
the system.



Fedora 31 is released. On Fedora 31, the module "foo" has a new
default stream "v1.1". When upgrading from Fedora 30 to Fedora 31,
Alice expects the package Foo she installed to be upgraded to version
1.1, because that's what would have happened if it was provided as a
package from the non-modular repositories.



Use Case 2:

On Fedora 30, user Bob runs

yum enable foo:v1.0

In this case, the "v1.0" stream of the "foo" module has a dependency
on the "v2.4" stream of the "bar" module. So when enabling "foo:v1.0",
the system also implicitly enables "bar:v2.4".



Fedora 31 is released. On Fedora 31, the module stream "foo:v1.0" now
depends on "bar:v2.5" instead of "bar:v2.4". The user, caring only
about "foo:v1.0" would expect the upgrade to complete, adjusting the
dependencies as needed.


At Flock and other discussions, we've generally come up with a
solution, but it's not yet recorded anywhere. I'm sending it out for
wider input, but this is more or less the solution we intend to run
with, barring someone finding a severe flaw.

Proposed Solution:

What happens today is that once the stream is set, it is fixed and
unchangeable except by user decision. Through discussions with UX
folks, we've more or less come to the decision that the correct
behavior is as follows:

* The user's "intention" should be recorded at the time of module
enablement. Currently, module streams can exist in four states:
"available, enabled, disabled, default". We propose that there should
be two additional states (names TBD) representing implicit enablement.
The state "enabled" would be reserved for any stream that at some
point was enabled by name. For example, a user who runs `yum install
freeipa:DL1` is making a conscious choice to install the DL1 stream of
freeipa. A user who runs `yum install freeipa-client` is instead
saying "give me whatever freeipa-client is the default".

* The state `dep_enabled` would be set whenever a stream becomes
enabled because some other module stream depended on it. This state
must be entered only if the previous state was `default` or
`available`. (We don't want `enabled` or `disabled` streams being able
to transition to this state.)

* The state `default_enabled` would be set whenever a stream becomes

Re: Modularity vs. libgit

[Daniel Mach]

Dne 14. 06. 19 v 6:23 Samuel Sieb napsal(a):
After reading the bug report and the discussions, I still don't 
understand why dnf is complaining about a conflict with packages 
(modules?) that are not installed and are not even trying to be 
installed.  Can someone explain that?


The dependency resolution is done on 2 levels.
RPM dependency resolution still works the same even with modules.

What's different on a modular system is a pre-processing step that 
resolves modules and filters out RPMs that do not match with 
enabled/default streams.


What makes it even more complicated is stream expansion:
https://docs.fedoraproject.org/en-US/modularity/architecture/stream-expansion/

Some modules get build several times against multiple dependency sets.
These dependency sets are captured as Context field in module metadata.
DNF has to resolve the contexts and pick the right ones based on enabled 
streams. This is done for *every* module version (not just the latest) 
in order to compute which RPMs will be available to the user and which 
not. And this causes the conflicts you were asking about.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

[Daniel Mach]

Dne 31. 05. 19 v 2:15 Pavel Raiskup napsal(a):

On Thursday, May 30, 2019 10:38:25 AM CEST Miroslav Suchý wrote:

Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a):

If we did this, wouldn't it make it very difficult to use tools like
mock on RHEL / CentOS 7 to build for Fedora 3x?


Speaking of Mock:
Either the RPM on host need to understand the new format/compression **or**
the packages in @buildsys group (including transitional deps) have to be in
old format - then you can build for Fedora 3x using bootstrap feature.


I need to underline this, it would be really really really bad if we were
not able to --installroot fedora chroots at least on RHEL 8.  How likely
is a backport of zstd support into RPM in EL7+?
RHEL 8 should be quite easy - get zstd package into RHEL and recompile 
rpm with zstd support.


RHEL 7 is a different story. The patch doesn't apply directly and a 
backport would be needed.


Panu,
how difficult it would be to backport the zstd support to RHEL 7 RPM?



Regarding @buildsys group and compat compression;  if we were OK to use
`mock --bootstrap-chroot`, we could limit the package compatibility set
(not really subset) to dnf + dnf-utils + deps (see dnf_install_command in
site-defaults.cfg).  But TBH I don't view this idea as feasible/maintainable
solution, "Requires:" do change all the time...

Another slightly more realistic way around would be to not --installroot
the bootstrap chroot in mock, but rely on some distributed "bootstrap"
root cache tarball (a standard, safe way) instead.


Both of them would be painful.


Typo: "having none of them would be painful", and very likely to happen.

Pavel


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

[Daniel Mach]

Dne 31. 05. 19 v 2:15 Pavel Raiskup napsal(a):

On Thursday, May 30, 2019 10:38:25 AM CEST Miroslav Suchý wrote:

Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a):

If we did this, wouldn't it make it very difficult to use tools like
mock on RHEL / CentOS 7 to build for Fedora 3x?


Speaking of Mock:
Either the RPM on host need to understand the new format/compression **or**
the packages in @buildsys group (including transitional deps) have to be in
old format - then you can build for Fedora 3x using bootstrap feature.


I need to underline this, it would be really really really bad if we were
not able to --installroot fedora chroots at least on RHEL 8.  How likely
is a backport of zstd support into RPM in EL7+?
RHEL 8 should be quite easy - get zstd package into RHEL and recompile 
rpm with zstd support.


RHEL 7 is a different story. The patch doesn't apply directly and a 
backport would be needed.


Panu,
how difficult it would be to backport the zstd support to RHEL 7 RPM?



Regarding @buildsys group and compat compression;  if we were OK to use
`mock --bootstrap-chroot`, we could limit the package compatibility set
(not really subset) to dnf + dnf-utils + deps (see dnf_install_command in
site-defaults.cfg).  But TBH I don't view this idea as feasible/maintainable
solution, "Requires:" do change all the time...

Another slightly more realistic way around would be to not --installroot
the bootstrap chroot in mock, but rely on some distributed "bootstrap"
root cache tarball (a standard, safe way) instead.


Both of them would be painful.


Typo: "having none of them would be painful", and very likely to happen.

Pavel


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

[Daniel Mach]

Dne 30. 05. 19 v 17:14 Peter Robinson napsal(a):

What about constrained systems where there's limited CPU, that could
be a container or a low end cloud instance without any guarantee of
resources or a Raspberry Pi.


Raspberry Pi is hard to measure. I'm getting quite random results due to 
IO, I blame the on-board microSD card controller.


I tested following package set (podman + deps):
containernetworking-plugins-0.7.4-2.fc30.aarch64.rpm
containers-common-0.1.36-9.dev.gitd93a581.fc30.aarch64.rpm
criu-3.12-11.fc30.aarch64.rpm
fuse3-libs-3.5.0-1.fc30.aarch64.rpm
fuse-overlayfs-0.3-10.dev.gita7c8295.fc30.aarch64.rpm
libbsd-0.9.1-3.fc30.aarch64.rpm
libnet-1.1.6-17.fc30.aarch64.rpm
ostree-libs-2019.2-1.fc30.aarch64.rpm
podman-1.3.1-1.git7210727.fc30.aarch64.rpm
protobuf-c-1.3.1-2.fc30.aarch64.rpm
runc-1.0.0-93.dev.gitb9b6cc6.fc30.aarch64.rpm
slirp4netns-0.3.0-2.git4992082.fc30.aarch64.rpm
tar-1.32-1.fc30.aarch64.rpm

Then I re-compressed the RPMs by running:
rpmrebuild --define "%_binary_payload w19.zstdio" -p 

xz.2 size: 27M
zstd.19 size: 22M

Then I ran installs into ramdisk:
sync; echo 3 > /proc/sys/vm/drop_caches; time rpm -Uvh * --force 
--nodeps --stats --noscripts --root=/tmp/foo


xz: 10.9s
zstd: 4.7s

When I installed RPMs on the system (micro SD card), it took approx. 
1m30s in both cases, but it's really giving quite random results due to IO.


Zstd performs slightly better than xz, but users can hardly see any 
difference, because of IO. Good thing is that it's almost equal and 
doesn't degrade performance on low end hardware.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

[Daniel Mach]

Dne 30. 05. 19 v 17:14 Peter Robinson napsal(a):

What about constrained systems where there's limited CPU, that could
be a container or a low end cloud instance without any guarantee of
resources or a Raspberry Pi.


Raspberry Pi is hard to measure. I'm getting quite random results due to 
IO, I blame the on-board microSD card controller.


I tested following package set (podman + deps):
containernetworking-plugins-0.7.4-2.fc30.aarch64.rpm
containers-common-0.1.36-9.dev.gitd93a581.fc30.aarch64.rpm
criu-3.12-11.fc30.aarch64.rpm
fuse3-libs-3.5.0-1.fc30.aarch64.rpm
fuse-overlayfs-0.3-10.dev.gita7c8295.fc30.aarch64.rpm
libbsd-0.9.1-3.fc30.aarch64.rpm
libnet-1.1.6-17.fc30.aarch64.rpm
ostree-libs-2019.2-1.fc30.aarch64.rpm
podman-1.3.1-1.git7210727.fc30.aarch64.rpm
protobuf-c-1.3.1-2.fc30.aarch64.rpm
runc-1.0.0-93.dev.gitb9b6cc6.fc30.aarch64.rpm
slirp4netns-0.3.0-2.git4992082.fc30.aarch64.rpm
tar-1.32-1.fc30.aarch64.rpm

Then I re-compressed the RPMs by running:
rpmrebuild --define "%_binary_payload w19.zstdio" -p 

xz.2 size: 27M
zstd.19 size: 22M

Then I ran installs into ramdisk:
sync; echo 3 > /proc/sys/vm/drop_caches; time rpm -Uvh * --force 
--nodeps --stats --noscripts --root=/tmp/foo


xz: 10.9s
zstd: 4.7s

When I installed RPMs on the system (micro SD card), it took approx. 
1m30s in both cases, but it's really giving quite random results due to IO.


Zstd performs slightly better than xz, but users can hardly see any 
difference, because of IO. Good thing is that it's almost equal and 
doesn't degrade performance on low end hardware.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

[Daniel Mach]

Dne 30. 05. 19 v 14:57 Kevin Kofler napsal(a):

Jason L Tibbitts III wrote:


"BC" == Ben Cotton  writes:

BC> * The recommended compression level is 19. The builds will take
BC> longer, but the additional compression time is negligible in the
BC> total build time and it pays off in better compression ratio than xz
BC> lvl2 has.

That seems different than other results I've seen.  According to the
wikipedia page (https://en.wikipedia.org/wiki/Zstandard) and the
references therein, Ubuntu found that zstd level 19 was faster but with
poorer compression when compared with xz level 2 (which is the same
level that we use now).
I haven't recompiled all Fedora, hoping that the package set I used 
(Livecd RPMs) is a good sample. Thanks for pointing this out.




If the smallest size is the goal, wouldn't it be worth trying to just use a
higher xz level instead?
That would increase the compression ratio, but I don't know the impact 
on compression and decompression times and also required memory for both 
operations.
On modern system with NVMEs, CPU seems to be the bottleneck and it might 
get worse if we increase xz compression level.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

[Daniel Mach]

Dne 30. 05. 19 v 8:39 Igor Gnatenko napsal(a):
Last time I was about to propose this in F29, I did mass-rebuild myself 
and while decompressing was faster in most of the cases, the size was 
definitely worse. So definitely "Lower bandwidth on mirrors if we choose 
the highest compression level" is under the question.
My current observation is that compression ratio differs per package, 
sometimes xz.2 wins, sometimes it's zstd.19.
The data I initially picked compresses better with zstd, while 
recompiled RPMs that are installed in fedora:30 docker image have almost 
equal size.


BTW, which compression level did you use?
Could you share some of your observations and stats if you still have them?



I think before approving such changes, owners need to do mass rebuilds 
on their own and provide a graph of changes in size between original 
compression format and new one(s).

Doing that equals to the mass rebuild.
I'd rather do an analysis and if the numbers look sane, I'd prefer doing 
a mass rebuild in a side tag so we can use the builds if the results are 
sane. Hammering koji with so many scratch builds doesn't sound right to me.


Just saying it works better on Firefox doesn't sound to me like the way 
to go.

Firefox was an example.
The other table shows real-life data based on RPMs available on Fedora 
LiveCD.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

[Daniel Mach]

Dne 30. 05. 19 v 0:05 Neal Gompa napsal(a):


I'm pretty sure this would break DeltaRPMs, since none of the drpm
software has been updated to handle zstd compression. Neither drpm nor
deltarpm handle it today.


Thanks for heads-up. We'll look into it and provide a fix soon.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

[Daniel Mach]

Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a):

On Wed, May 29, 2019 at 4:20 PM Ben Cotton  wrote:


https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression

= Switch RPMs to zstd compression =

== Summary ==
Binary RPMs are currently compressed with xz level 2.
Switching to zstd would increase decompression speed significantly.

== Owner ==
* Name: [[User:dmach| Daniel Mach]]
* Email: dm...@redhat.com

== Detailed Description ==
* The change requires setting a new compression algorithm in rpm
macros. Then a mass rebuild of all packages is required.


The gcc team often does mass rebuilds on the side prior to updating
gcc in Fedora.  Would it be possible to do the same or leverage their
rebuild work with the default changed in RPM to see what the true
overall savings is?  That would get us a lot more data to see if it's
truly going to benefit the distro in terms of size and installation
speed.


I rebuilt the packages that are available in fedora:30 docker image:
https://copr.fedorainfracloud.org/coprs/dmach/fedora-zstd/

The overall size is roughly equal to xz compressed RPMs.

It's not comparable with the whole Fedora repo, but it's a good start.
I can build more packages if a larger sample is needed.




* The macro for setting the compression is: %define _binary_payload w19.zstdio
* The recommended compression level is 19. The builds will take
longer, but the additional compression time is negligible in the total
build time and it pays off in better compression ratio than xz lvl2
has.
* SRPM payload compression should stay at gzip (there's almost no
benefit in changing the compression, because SRPM's contents is
compressed already)

=== Use case: Firefox installation ===
I rebuilt firefox-66.0.5-1.fc30 with zstd level19.
Then I compared installation times with the original (xz compressed) package:

{| class="wikitable"
|-
! Compression !! Target File System !! Time
|-
| xz level 2  || tmpfs || 8s
|-
| xz level 2 || ext4 on nvme || 11s
|-
| zstd level 19  || tmpfs || 2s
|-
| zstd level 19  ||  ext4 on nvme || 4s
|-
|}


=== Comparison of compression algorithms and levels ===
Following table shows '''cpio''' and '''compressed cpio''' extraction
times into a tmpfs. Actual times in decompressing RPMs will differ due
to extracting on an actual disk and also some overhead in the RPM tool
(checks, scriptlets).

{| class="wikitable"
|-
! Compression   !! Level!! Size B   !! Size GiB
  !! Compression time !! Compression time, 4 threads  !!
Decompression time   !! Comment
|-
| CPIO  || -|| 5016785692   || 4,7
  || -|| -|| -
   ||
|-
| xz|| 2|| 1615017616   || 1,6
  || 9m55s|| -|| 1m36s
   || slow decompression
|-
| pxz   || 2|| 1631869880   || 1,6
  || -|| 6m11s|| 1m38s
   || slow decompression
|-
| gzip  || 9|| 2086354992   || 2,0
  || 10m23s   || -|| 31s
   || insufficient compression ratio
|-
| bzip2 || 9|| 1889161565   || 1,8
  || 8m   || -|| 2m50s
   || very slow decompression; compression ratio could be
better
|-
| zstd  || 3|| 1913536587   || 1,8
  || 31s  || 29s  || 6,5s
   ||
|-
| zstd  || 10   || 1737928978   || 1,7
  || 3m27s|| 2m34s|| 6,3s
   ||
|-
| zstd  || 15   || 1717303256   || 1,7
  || 9m37s|| 6m34s|| 6,3s
   || identical compression speed to xz; fast decompression;
slightly worse compression ratio than xz
|-
| zstd  || 17   || 1635525492   || 1,6
  || 16m16s   || 11m20s   || 6,7s
   ||
|-
| zstd  || 19   || 1575843696   || 1,5
  || 24m2s|| 18m55s   || 7,7s
   ||
|-
|}

== Benefit to Fedora ==
* Faster installations/upgrades of user systems
* Faster koji builds (installations in build roots)
* Faster container builds
* Lower bandwidth on mirrors if we choose the highest compression level

== Scope ==
* Proposal owners: submit a patch to redhat-rpm-config
* Other developers: redhat-rpm-config maintainer: include the patch
and make a new build
* Release engineering: [https://pagure.io/releng/issue/8345 #8345]
mass rebuild is needed

== Upgrade/compatibility impact ==
* RPM in Fedora supports zstd compression already (from Fedora 28,
rpm-4.14.0-0.rc2.5.fc28). No impact on Fedora users is expected.
* Fedora <= 27 and some other distros will not be able to decompress
zstd-compressed RPMs.


I

Re: dnf history - change in how rpmdb checksum is computed

[Daniel Mach]

On Thu, Jul 19, 2018 at 12:06 AM, Neal Gompa  wrote:

> On Wed, Jul 18, 2018 at 6:02 PM Daniel P. Berrangé 
> wrote:
> >
> > On Wed, Jul 18, 2018 at 03:24:54PM +0200, Daniel Mach wrote:
> > > Hi everyone,
> > > The DNF team is currently reviewing DNF compatibility with YUM 3 and
> we'd
> > > like to get feedback on this one:
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1120253
> > >
> > > rpmdb checksum is a checksum of all installed RPMs
> > > It has no cryptographical value, it's just an unique ID of RPMs on a
> system
> > > before and after each transaction and it's used in dnf history info
> and dnf
> > > history list.
> > > If checksums of 2 following transactions do not match, DNF indicates
> that.
> > > This happens if a user installs an RPM by hand via rpm command.
> > >
> > > Then `dnf history list` looks like:
> > >  2 | install bar | 2018-01-01 02:00 | Install|2  <
> > >  1 | install foo | 2018-01-01 01:00 | Install|7 >
> > > the "<" and ">" characters indicate discontinuity in rpmdb hashes
> > >
> > > Here's the question:
> > > DNF computes the checksum from RPM N-E:V-R.A
> > > while YUM computed it from E:N-V-R.A
> > >
> > > We'd like to change the behavior to be compatible with YUM again.
> > > This would create 1 discontinuity in rpmdb checksums in the history,
> > > because from that point a new algorithm will be used.
> > >
> > > Are there any concerns about such change?
> > > I believe that >90% users wouldn't notice anything as it's related to
> the
> > > history database only.
> >
> > What's the benefit in changing to be compatible with YUM as opposed
> > to stickin with current alogorithm ?
> >
> > Surely if we don't change it, even fewer users will notice that DNF's
> > behaviour is different from YUM's, since DNF has been the default for
> > many releases now.
> >
> > I could understand the motiviation to stay compatible with YUM if we
> > were only just about to switch Fedora from YUM to DNF, but time is
> > way in the past now. Shouldn't we optimize for the fact that DNF is
> > the more widely deployed & used tool, and thus not worry about
> > YUM compatibility in respect of the history DB ?
> >
>
> From my point of view, I considered YUM's rendering of the NEVRA to be
> very weird. Personally, I'd rather see us keep the DNF behavior for
> rendering NEVRA rather than switch to YUM's ENVRA.
>
> That's right, we definitely don't want to go back to ENVRA anywhere in the
UI or API.
The ENVRA format would be only an implementation detail in the function
that computes the checksum.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/DUFU46D7NU3AFU3ZEP4ES6MJRIUVZD24/

Re: dnf history - change in how rpmdb checksum is computed

[Daniel Mach]

On Wed, Jul 18, 2018 at 5:13 PM, Ben Cotton  wrote:

> On Wed, Jul 18, 2018 at 11:02 AM Daniel Mach  wrote:
>
> > If a user migrates from RHEL 7 to the next version of RHEL (or CentOS),
> > there will be continuity in used algorithm and history db checksums.
> > It's important to some enterprise customers to keep the history db in a
> good shape.
> > Fedora users don't care about that much in general.
> >
> This makes sense. Let me ask from another angle: does Fedora lose
> anything from not using the current dnf history algorithm (apart from
> the discontinuity when we switch)? Would it make sense to have that be
> a configurable option where Fedora defaults to the dnf model and RHEL
> defaults to the yum model or is it essentially a cosmetic difference?
>
To me, it's more a cosmetic difference and I don't think it deserves a
configurable option for switching the behavior.
It is more important to unify the behavior, document it and cover with
tests.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/GGU7CFCTBGEVJOTD5TZFQTGS55P3NWUP/

Re: dnf history - change in how rpmdb checksum is computed

[Daniel Mach]

>
> What's the benefit in changing to be compatible with YUM as opposed
> to stickin with current alogorithm ?
>
If a user migrates from RHEL 7 to the next version of RHEL (or CentOS),
there will be continuity in used algorithm and history db checksums.
It's important to some enterprise customers to keep the history db in a
good shape.
Fedora users don't care about that much in general.


> Surely if we don't change it, even fewer users will notice that DNF's
> behaviour is different from YUM's, since DNF has been the default for
> many releases now.
>
> I could understand the motiviation to stay compatible with YUM if we
> were only just about to switch Fedora from YUM to DNF, but time is
> way in the past now. Shouldn't we optimize for the fact that DNF is
> the more widely deployed & used tool, and thus not worry about
> YUM compatibility in respect of the history DB ?
>
Unfortunately RHEL knows nothing about DNF yet
and it's YUM compatibility what matters.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/JYCKOFC223P5RADHMZOBNBA5L2V6EGYS/

dnf history - change in how rpmdb checksum is computed

[Daniel Mach]

Hi everyone,
The DNF team is currently reviewing DNF compatibility with YUM 3 and we'd
like to get feedback on this one:
https://bugzilla.redhat.com/show_bug.cgi?id=1120253

rpmdb checksum is a checksum of all installed RPMs
It has no cryptographical value, it's just an unique ID of RPMs on a system
before and after each transaction and it's used in dnf history info and dnf
history list.
If checksums of 2 following transactions do not match, DNF indicates that.
This happens if a user installs an RPM by hand via rpm command.

Then `dnf history list` looks like:
 2 | install bar | 2018-01-01 02:00 | Install|2  <
 1 | install foo | 2018-01-01 01:00 | Install|7 >
the "<" and ">" characters indicate discontinuity in rpmdb hashes

Here's the question:
DNF computes the checksum from RPM N-E:V-R.A
while YUM computed it from E:N-V-R.A

We'd like to change the behavior to be compatible with YUM again.
This would create 1 discontinuity in rpmdb checksums in the history,
because from that point a new algorithm will be used.

Are there any concerns about such change?
I believe that >90% users wouldn't notice anything as it's related to the
history database only.

thanks,
Daniel
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/YOYJEKGZ3N4Q67TVOQ6MHBK37RESBVAQ/

Re: F29 Self Contained Change: Deprecate YUM 3

[Daniel Mach]

On Thu, Jul 5, 2018 at 9:45 PM, Kevin Fenzi  wrote:

> On 06/27/2018 03:18 AM, Lubomír Sedlář wrote:
> > Removing Yyum would mean that there will no longer be /usr/bin/pungi
> > available in Fedora. This is not a problem for any work done by release
> > engineering, but it is still used by people creating spins.
> >
> > So this is a call to action: if anyone wants to continue using it, now
> > is the time to come up and port it to DNF (and Python 3).
>
> Sorry for my delay in replying here... been busy. ;(
>
> Looking at the list of things that would go away (from the wiki page):
>
> cobbler
> ddiskit
> diskimage-builder
> dlrn
> dnf-plugins-core
>
> ? dnf-plugins-core is kind of important, is this a false positive?
> Or does it mean python2-dnf-plugin*?
>
Yes, it's python2-dnf-plugin-migrate.
The list I provided contains packages from koji/dist-git/bugzilla
perspective (srpm names).

>
> fusioninventory-agent
> grinder
> imgbased
> kiwi
> koji
>
> koji is kinda important. I think this is meaning python2-koji?
> I would hope python3-koji/koji stays around?
>
ditto

>
> koji-containerbuild
>
> We like containers these days, don't we?
>
> libtaskotron
>
> And testing?
>
> lpf
> mach
> mash
> mirrormanager
>
> And mirrors (this one isn't as important as we run mirrormanager on
> rhel, but still)
>
> nagios-plugins-check-updates
> osc
> perl-Fedora-Rebuild
> plague
> pulp-rpm
> repo_manager
> repoview
> retrace-server
>
> This also runs on rhel, but sooner or later will need porting.
>
> rpm-ostree-toolbox
> sigul
>
> This is kinda important.
>
> snake
> system-config-kickstart
> yum-axelget
> yum-rhn-plugin
> yum-updatesd
>
> So, I'm personally not too keen on this at this point. I suppose if it
> happens then we will need to maintain/keep a fork of yum3 in
> infrastructure for tools, at which point it might be best to just keep
> it in the distro.
>
> I don't know the porting plans for all the above stuff, but I would
> personally really prefer retiring only after they were ported or at
> least we know there's a plan for them.
>
> If the critical packages don't get ported in time, we can postpone
removing yum to the next release.
I think the hard deadline is when Python 2 gets removed from the distro
(depending on Python 2 indicates
that some packages are not Python 3 ready yet).


> kevin
>
>
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.
> fedoraproject.org/message/4ANJI5BPX2RYPNXNGJKGHQCXIYRUKIKS/
>
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/4Z7BY6GDP4BN4V5NRUPXD5FOGUVWYY5U/

Re: F29 Self Contained Change: Deprecate YUM 3

[Daniel Mach]

On Thu, Jun 28, 2018 at 1:35 AM, Adam Williamson  wrote:

> On Wed, 2018-06-27 at 18:18 -0400, Josh Boyer wrote:
> > On Wed, Jun 27, 2018 at 4:30 PM Adam Williamson
> >  wrote:
> > >
> > > On Wed, 2018-06-27 at 16:25 -0400, Matthew Miller wrote:
> > > > On Wed, Jun 27, 2018 at 02:54:07PM +0200, Björn Persson wrote:
> > > > > > IMHO deprecate != remove, but rather mark for removal in some
> next release.
> > > > > > Should the change be called differently?
> > > > >
> > > > > Especially since Yum has been called "yum-deprecated" for several
> > > > > releases already.
> > > >
> > > > How about "Replace Yum 3 with Yum 4, powered by DNF"? This would
> bring
> > > > us in line with what's happening in the Enterprise Linux space.
> > > >
> > > > (See
> > > > https://wiki.centos.org/SpecialInterestGroup/
> ConfigManagementSIG/YUM4)
> > >
> > > But in Fedora land, we've spent several years selling the message "yum
> > > is gone and replaced with this new thing called dnf". It would be
> > > rather confusing to suddenly start selling the message "oh hey yum is
> > > back only now it's sort of dnf but sort of not dnf".
> >
> > It's still dnf.  In fact, I believe /usr/bin/dnf would even still
> > exist.  However, dnf has come significantly closer to yum
> > functionality since it was first introduced and reuniting isn't a bad
> > idea.
> >
> > I understand where you're coming from, but I think we should take the
> > opportunity to correct now.  We (and I do mean we as someone that
> > pushed for not calling it yum) had valid reasons to separate it in the
> > past, but those reasons are becoming increasingly invalid.  Sticking
> > with the dnf name is going to become a forced split going forward for
> > little benefit.  I'm happy to eat my own words and say we should
> > probably focus around a single package manager name at this point.
> >
> > > It's different from the EL situation because EL never really had the
> > > "dnf is the new thing" phase. If you're going from EL 7 to The Next EL
> > > you're just going from yum 3 to "yum 4".
> >
> > Yeah, but if you play in both spaces continuing to call it "dnf" in
> > Fedora and "yum4" in EL is forcing a mental break that doesn't really
> > need to be there.
>
> So I may have missed the latest shiny plans here - I thought the plan
> was that dnf would provide a 'yum' CLI command which was as close as
> possible to compatible with yum 3, but *also* provide a 'dnf' CLI
> command which was more like the 'current' dnf CLI in Fedora. Is that
> still the case? Or is there just going to be one true CLI command now?
>
> DNF shouldn't diverge from YUM just "because we can".
We're fixing some obvious differences that weren't introduced for any good
reason.

There will be no special compat layer just a yum -> dnf symlink.
If the compatibility is preserved to sufficient level, we believe it's a
better
option than to have 2 executables with different behavior.

The long-term priority is to make DNF command-line interface and behavior
consistent
and in such cases, DNF must diverge from YUM3 behavior and insisting on 100%
compatibility would block usability improvements and evolution in general.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/BLKXTF4K3WZPM2ZHHDAWTBKJIMMUXXG3/

Re: F29 Self Contained Change: Deprecate YUM 3

[Daniel Mach]

On Wed, Jun 27, 2018 at 2:23 PM, Sam Varshavchik 
wrote:

> Jan Kurik writes:
>
> = Proposed Self Contained Change: Deprecate YUM 3 =
>> https://fedoraproject.org/wiki/Changes/Deprecate_YUM_3
>>
>>
>> Owner(s):
>>   * Daniel Mach 
>>
>>
>> Remove yum (v3) and all related packages from Fedora.
>>
>>
>> == Detailed description ==
>> Remove packages from the distribution:
>> * createrepo
>>
>
> We should have createrepo_c install a symlink, and have an "Obsoletes:
> createrepo" so that this gets taken care of by a normal upgrade.
>
> Yes, we'll provide compat Provides and symlinks to createrepo_c and yum so
many command line users won't see any big difference.
I've updated the document with the 'Compatibility' section.
Thanks!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/NMLNEFKAJKPCER5KDS45SORDFYF4V2AT/

Announcing DNF 3 development

[Daniel Mach]

We are pleased to announce that development of DNF 3 has started. This
version is focused on performance improvements, new API and consolidating
the whole software management stack.

Please read more details on our blog:
https://rpm-software-management.github.io/announcement/2018/03/22/dnf-3-announcement/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Install Fedora by hand ... feeling brave :-)

[Daniel Mach]

Dne 7.6.2015 v 16:12 David Timms napsal(a):

Hi, I'm keen to install F22 on a laptop (shared with windows 7), but the
installer bugs out due to problems in libparted when reading my disk
(bz: 1223111):
https://bugzilla.redhat.com/show_bug.cgi?id=1223111

The machine boots and runs the live iso via usb, but also can't do
install to the HD due to the above.

So, what would it take to set up Fedora by hand ?

My first go is:
boot live iso to runlevel 2
login: root

mount /dev/sda6 /mnt
mount /dev/sda5 /mnt/boot
mount /dev/sda7 /mnt/home
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/proc
mount -t tmpfs  tmpfs  /mnt/tmp ???

dnf --installroot=/mnt install filesystem kernel firefox grub2  grubby
(workstation stuff)

chroot /mnt
passwd

dnf reinstall kernel (to make initrd creation work.)

useradd myuser
passwd myuser
usermod -G wheel ??

grub2-install /dev/sda
grub2-mkconfig -o /boot/grub2/grub.cfg

What other things will I need to do / what does Anaconda do ?

My first effort looks like it may be hanging/dying on selinux parts; I
was able to /etc/selinux/config to turn is to DISABLED and add selinux=0
to kernel command line.

Since all my files are installed without SE context, boot fails with SE
enforcing. If SE disabled then restorecon etc don't work. What's the
steps, to bootstrap selinux ?

I imagine someone has done / documented this before, but google search
not fruitful.



I was installing my laptop the same way you're describing.
It worked fine for for me until Fedora 18.
Unfortunately something changed and I installed Fedora 19 using Anaconda again 
(had no time to debug).

I think the only missing steps are to create /etc/fstab and make selinux work 
by running `touch /.autorelabel`

- daniel

--
Daniel Mach dm...@redhat.com
Release Engineering, Red Hat
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: sorting yum/dnf metadata and metadata diffs

[Daniel Mach]

Hi,
there's been some work in progress already:
https://bugzilla.redhat.com/show_bug.cgi?id=850896

Proof-of-concept code (to be merged into dnf/createrepo_c in the future):
https://github.com/Tojaj/DeltaRepo


The idea behind that is simple:
* create deltas as small repos on server
* download deltas on client
* do in-memory mergerepo on client
  (or cache it on disk if it makes sense)


I consider this approach better than making diffs,
especially because it's simple, clean and it can work with any repo format 
(sqlite, xml or mix of both).


- daniel

Dne 13.2.2015 v 08:11 Casey Jao napsal(a):

How feasible would it be to keep the listings in primary.xml and
filelists.xml sorted by package name and arch? Doing so could open the
door to simple and efficient diffs of repository metadata.

I recently ran some quick tests using python and elementtree. While the
F21 primary.xml files from 2/7 and 2/9 both weigh around 2.6M compressed
and ~18M uncompressed, sorting them and running a simple line-by-line
comparison revealed a diff of ~500K, which compressed down to ~70K. A
similar procedure on the 8M filelists.xml yielded a diff which
compressed to ~200K.

Those two are by far the largest metadata files. If the observed
improvements are typical, then keeping those files in order and hosting
the diffs between the present and the previous few days (and modifying
dnf to look for those diffs) could substantially reduce the amount of
data that users must download every time a repository is updated, which
for a fast-moving OS like Fedora could happen nearly every day.





--
Daniel Mach dm...@redhat.com
Release Engineering, Red Hat
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct