Re: rpm overwriting config files again
On Jun 9, 2015, at 5:50 PM, Elan Ruusamäe wrote: > ping jbj? > Yes? I don't think your change to %config handling is worth the effort and disruption to the existing %config algorithm. Grep the code for FA_ALTNAME (for .rpmnew) and FA_BACKUP (for .rpmorig/.rpmsave) handling, patch as you wish. The change will likely be 1-3 lines of logic to set FA_BACKUP rather than FA_CREATE under some set of conditions. 73 de Jeff ___ pld-devel-en mailing list pld-devel-en@lists.pld-linux.org http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: rpm overwriting config files again
ping jbj?
On 30.05.2015 10:30, Elan Ruusamäe wrote:
On 29.05.2015 17:43, Jeffrey Johnson wrote:
If you wish to change the behavior to, say, create foo.rpmnew instead
of foo
when the new package file is marked %config(noreplace), then patch
lib/rpmfi.c to return FA_ALTNAME instead of FA_CREATE when marked
%config(noreplace) in lib/rpmfi.c rpmfiDecideFate()
The patch will likely be as simple as
-return FA_CREATE;
+return save;
i don't know what i'm doing wrong, but the fate is not decided for
commands.cfg file:
10:26:18 root[load: 0.13]@new-server ~# rpm -Uvv
nagios-nrpe-2.15-4.1.i686.rpm 2> inst.log
Preparing packages for installation...
Repackaging erased files...
nagios-nrpe-2.15-1.i686
Upgrading packages...
nagios-nrpe-2.15-4.1.i686
10:29:10 root[load: 0.01]@new-server ~# rpm -qf
/etc/nagios/nrpe.d/commands.cfg
nagios-nrpe-2.15-4.1.i686
10:26:22 root[load: 0.12]@new-server ~# grep -i fate inst.log
D: /etc/nagios/nrpe.cfg rpmfiDecideFate
besides your patch,
https://github.com/pld-linux/rpm/compare/config-noreplace-newfile
i added log when the method is invoked:
--- rpm-5.4.15/lib/rpmfi.c~ 2015-05-30 10:15:12.0 +0300
+++ rpm-5.4.15/lib/rpmfi.c 2015-05-30 10:21:23.644364230 +0300
@@ -677,6 +677,8 @@
struct stat sb;
int save = (newFlags & RPMFILE_NOREPLACE) ? FA_ALTNAME : FA_SAVE;
+ rpmlog(RPMLOG_DEBUG, D_("%s rpmfiDecideFate\n"), fn);
+
if (Lstat(fn, &sb)) {
/*
* The file doesn't exist on the disk. Create it unless the new
--
glen
___
pld-devel-en mailing list
pld-devel-en@lists.pld-linux.org
http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: rpm overwriting config files again
> On May 30, 2015, at 11:19 PM, Tomasz Pala wrote:
>
> On Fri, May 29, 2015 at 23:22:39 -0400, Jeffrey Johnson wrote:
>
>> You do realize that RPM5 embeds libgit2 sufficiently well to perform
>> a commit that git(1) understands?
>
> No... any docs? examples? How to use it? Can't find anything except
> rpmgitdebug, nothing appropriate in iosm, so I'm assuming this
> particular usage scenario is possible, but not yet available, is it?
>
libgit2 is embedded like all other embeddings as a macro expansion
%{git OPTIONS ARGS:BODY}
You can just as easily use %(git …) as %{git …};
There’s a hunk of code in rpmdb that will expand macros (and run git)
immediately after a header is added or deleted in an rpmdb. This is
currently how /var/lib/hrmib is populated for snmpd purposes. Files
could be managed much like etckeeper does as part of an
installation.
Meanwhile libgit2 hasn’t a stable AP: the code use to break monthly. I
am building rpm directly against HEAD of libgit2: that isn’t likely
going to be possible for any distro.
Meanwhile there’s no reason whatsoever that you can’t attempt
an etckeeper-like mechanism with a modest amount of scripting.
The harder problem for me is that RPM is expected to Just Work
in empty chroot’s, on bare metal hardware, and without git as
an install prereq. That is/was the reason for embedding libgit2.
>> Discusssing Newer! Better! Bestest!
>> %config handling, with (.rpmnew/.rpmorig/.rpmsave) handling is
>> so so so last century ...
>
> Indeed it is. I'm tired of handling all of this manually.
>
> With monotonically increasing versions of packages installed (i.e. only
> upgrades, no downgrades), this should be relatively simple:
> - keep some branch for original %configs (.rpmnew ones),
> - keep master branch for user-modified %configs (.rpmorig and .rpmsave),
> - try to git-merge changes on update (which would often fail, especially
> when updating very old package with multiple changes along the way -
> such files might be left for manual git merge/resolve).
>
> To properly support downgrades, rebase is needed to insert a file
> between older and newer one, and some tags to distinguish version.
>
git, not rpm, will be the tool to do rebasing etc. rpm can/will automate
some very simple steps much like etckeeper to handle %config files,
and perhaps add some options to do a diff with —query, or a merge
with —install.
>
> BTW, referring to the main subject:
> https://www.redhat.com/archives/rpm-list/2003-May/msg00426.html
>
*shrug* The %config handling in RPM was exactly as broken 12 years ago
as it is now. I see no important reason to change because it will lead to
several
years of questions and diagnoses simply because some change exists.
73 de Jeff
> --
> Tomasz Pala
> ___
> pld-devel-en mailing list
> pld-devel-en@lists.pld-linux.org
> http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
___
pld-devel-en mailing list
pld-devel-en@lists.pld-linux.org
http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: rpm overwriting config files again
On Fri, May 29, 2015 at 23:22:39 -0400, Jeffrey Johnson wrote: > You do realize that RPM5 embeds libgit2 sufficiently well to perform > a commit that git(1) understands? No... any docs? examples? How to use it? Can't find anything except rpmgitdebug, nothing appropriate in iosm, so I'm assuming this particular usage scenario is possible, but not yet available, is it? > Discusssing Newer! Better! Bestest! > %config handling, with (.rpmnew/.rpmorig/.rpmsave) handling is > so so so last century ... Indeed it is. I'm tired of handling all of this manually. With monotonically increasing versions of packages installed (i.e. only upgrades, no downgrades), this should be relatively simple: - keep some branch for original %configs (.rpmnew ones), - keep master branch for user-modified %configs (.rpmorig and .rpmsave), - try to git-merge changes on update (which would often fail, especially when updating very old package with multiple changes along the way - such files might be left for manual git merge/resolve). To properly support downgrades, rebase is needed to insert a file between older and newer one, and some tags to distinguish version. BTW, referring to the main subject: https://www.redhat.com/archives/rpm-list/2003-May/msg00426.html -- Tomasz Pala ___ pld-devel-en mailing list pld-devel-en@lists.pld-linux.org http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: rpm overwriting config files again
On 29.05.2015 17:43, Jeffrey Johnson wrote:
If you wish to change the behavior to, say, create foo.rpmnew instead of foo
when the new package file is marked %config(noreplace), then patch
lib/rpmfi.c to return FA_ALTNAME instead of FA_CREATE when marked
%config(noreplace) in lib/rpmfi.c rpmfiDecideFate()
The patch will likely be as simple as
- return FA_CREATE;
+ return save;
i don't know what i'm doing wrong, but the fate is not decided for
commands.cfg file:
10:26:18 root[load: 0.13]@new-server ~# rpm -Uvv
nagios-nrpe-2.15-4.1.i686.rpm 2> inst.log
Preparing packages for installation...
Repackaging erased files...
nagios-nrpe-2.15-1.i686
Upgrading packages...
nagios-nrpe-2.15-4.1.i686
10:29:10 root[load: 0.01]@new-server ~# rpm -qf
/etc/nagios/nrpe.d/commands.cfg
nagios-nrpe-2.15-4.1.i686
10:26:22 root[load: 0.12]@new-server ~# grep -i fate inst.log
D: /etc/nagios/nrpe.cfg rpmfiDecideFate
besides your patch,
https://github.com/pld-linux/rpm/compare/config-noreplace-newfile
i added log when the method is invoked:
--- rpm-5.4.15/lib/rpmfi.c~ 2015-05-30 10:15:12.0 +0300
+++ rpm-5.4.15/lib/rpmfi.c 2015-05-30 10:21:23.644364230 +0300
@@ -677,6 +677,8 @@
struct stat sb;
int save = (newFlags & RPMFILE_NOREPLACE) ? FA_ALTNAME : FA_SAVE;
+ rpmlog(RPMLOG_DEBUG, D_("%s rpmfiDecideFate\n"), fn);
+
if (Lstat(fn, &sb)) {
/*
* The file doesn't exist on the disk. Create it unless the new
--
glen
___
pld-devel-en mailing list
pld-devel-en@lists.pld-linux.org
http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: rpm overwriting config files again
On 30.05.2015 06:22, Jeffrey Johnson wrote: The alternative scenario is equally plausible: Sysadmin is trying to install software manually, lacks expertise in configuring. Eventually\ the sysadmin attempts install from rpm packages, mis-configuration is overwritten, and everything Just Work. Which isn't true if the newly installed configuration is renamed with a .rpmnew suffix. sysadmins installs software, and sees .rpmnew was created. this is common indication that config needs to be merged. there are plenty of existing scenarios that new software does not work until sysadmin merges .rpmnew config. -- glen ___ pld-devel-en mailing list pld-devel-en@lists.pld-linux.org http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: rpm overwriting config files again
On 30.05.2015 06:22, Jeffrey Johnson wrote: (aside) You do realize that RPM5 embeds libgit2 sufficiently well to perform a commit that git(1) understands? Discusssing Newer! Better! Bestest! %config handling, with (.rpmnew/.rpmorig/.rpmsave) handling is so so so last century ... no but i use etckeeper -- glen ___ pld-devel-en mailing list pld-devel-en@lists.pld-linux.org http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: rpm overwriting config files again
On May 29, 2015, at 3:25 PM, Elan Ruusamäe wrote: > On 29.05.2015 17:43, Jeffrey Johnson wrote: >> I won't make the change in RPM upstream because I believe it's >> more important to have consistent %config handling than it is to >> preserve unpackaged configuration files on upgrade when the >> new (but not the old) package has file content (which almost never happens). >> >> I can be convinced otherwise if there is a demonstrable consensus that a >> different >> behavior is needed and necessary. > the scenario is quite simple: > > some software loads config from conf.d dir (using *.conf glob), at some point > sysadmins put their configs there. > now rpm starts to put their files there too, it happens to be the same > filename, > now if the file already exists, it will ovewrite sysadmin changes. > > if it created .rpmnew, then the previous file (created by sysadmin) would be > used, and new config (from rpm), would be stored as .rpmnew > > overall config stays running, nothing is overwritten. > The alternative scenario is equally plausible: Sysadmin is trying to install software manually, lacks expertise in configuring. Eventually\ the sysadmin attempts install from rpm packages, mis-configuration is overwritten, and everything Just Work. Which isn't true if the newly installed configuration is renamed with a .rpmnew suffix. Enough already: you know well how RPM behaves with %confoig, and you are able to patch in whatever behevaior you choose: I told you how to patch in the behavior you desire. (aside) You do realize that RPM5 embeds libgit2 sufficiently well to perform a commit that git(1) understands? Discusssing Newer! Better! Bestest! %config handling, with (.rpmnew/.rpmorig/.rpmsave) handling is so so so last century ... *shrug* 73 de Jeff > -- > glen > > ___ > pld-devel-en mailing list > pld-devel-en@lists.pld-linux.org > http://lists.pld-linux.org/mailman/listinfo/pld-devel-en ___ pld-devel-en mailing list pld-devel-en@lists.pld-linux.org http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: rpm overwriting config files again
On 29.05.2015 17:43, Jeffrey Johnson wrote: I won't make the change in RPM upstream because I believe it's more important to have consistent %config handling than it is to preserve unpackaged configuration files on upgrade when the new (but not the old) package has file content (which almost never happens). I can be convinced otherwise if there is a demonstrable consensus that a different behavior is needed and necessary. the scenario is quite simple: some software loads config from conf.d dir (using *.conf glob), at some point sysadmins put their configs there. now rpm starts to put their files there too, it happens to be the same filename, now if the file already exists, it will ovewrite sysadmin changes. if it created .rpmnew, then the previous file (created by sysadmin) would be used, and new config (from rpm), would be stored as .rpmnew overall config stays running, nothing is overwritten. -- glen ___ pld-devel-en mailing list pld-devel-en@lists.pld-linux.org http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: rpm overwriting config files again
On May 29, 2015, at 4:28 AM, Elan Ruusamäe wrote: > On 29.05.2015 00:13, Jeffrey Johnson wrote: ... >> Was this behavior seen in a multiple package upgrade that you have snipped >> out? > nop. this was 1 package upgrade: > OK thanks for -vv output. > # rpm -q nagios-nrpe > nagios-nrpe-2.15-1.i686 > > # > /etc/nagios/nrpe.d/commands.cfg > > # rpm -qf /etc/nagios/nrpe.d/commands.cfg > file /etc/nagios/nrpe.d/commands.cfg is not owned by any package > So an empty commands.cfg is generated. > # rpm -Uvv nagios-nrpe-2.15-4.1.i686.rpm > (aside) Nice! PLD appears to be trying to eliminate LOOPS. > D: ignore package name relation(s) [0] QtGui -> Qt3Support > D: ignore package name relation(s) [1] QtGui -> QtSvg > ... > D: == tsorting packages (order, #predecessors, #succesors, tree, > Ldepth, Rbreadth) > D: 001100 +nagios-nrpe-2.15-4.1.i686 > D: == successors only (41485 bytes) > D: 110110 -nagios-nrpe-2.15-1.i686 ... > > D: fini 100640 1 ( 0, 72)6132 /etc/nagios/nrpe.cfg;55682259 > altname > warning: /etc/nagios/nrpe.cfg created as /etc/nagios/nrpe.cfg.rpmnew > D: fini 040750 2 ( 0, 72) 0 /etc/nagios/nrpe.d > D: fini 100640 1 ( 0, 72)1828 > /etc/nagios/nrpe.d/commands.cfg;55682259 ... > > D: fini 100640 1 ( 0, 0)7215 /etc/nagios/nrpe.cfg skip > > # ls -l /etc/nagios/nrpe.d/commands.cfg > -rw-r- 1 root nagios 1828 May 25 16:03 /etc/nagios/nrpe.d/commands.cfg > > # rpm -q rpm > rpm-5.4.15-20.i686 > > # rpm --version > rpm (RPM) 5.4.15 So the problem comes down to user expectations of behavior when the new package has a file marked %config(noreplace). The observed behavior is exactly what I personally expect of RPM (and is what has been implemented all this century AFAIK). If you wish to change the behavior to, say, create foo.rpmnew instead of foo when the new package file is marked %config(noreplace), then patch lib/rpmfi.c to return FA_ALTNAME instead of FA_CREATE when marked %config(noreplace) in lib/rpmfi.c rpmfiDecideFate() The patch will likely be as simple as - return FA_CREATE; + return save; I won't make the change in RPM upstream because I believe it's more important to have consistent %config handling than it is to preserve unpackaged configuration files on upgrade when the new (but not the old) package has file content (which almost never happens). I can be convinced otherwise if there is a demonstrable consensus that a different behavior is needed and necessary. 73 de Jeff ___ pld-devel-en mailing list pld-devel-en@lists.pld-linux.org http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: rpm overwriting config files again
On 29.05.2015 00:13, Jeffrey Johnson wrote: My guess (similar to your previous claim of "overwritten") is that package order ended up as erase-before-install as part of a multiple package upgrade. yes, pld rpm 4.5 got some (your) patch there eventually. Was this behavior seen in a multiple package upgrade that you have snipped out? nop. this was 1 package upgrade: # rpm -q nagios-nrpe nagios-nrpe-2.15-1.i686 # > /etc/nagios/nrpe.d/commands.cfg # rpm -qf /etc/nagios/nrpe.d/commands.cfg file /etc/nagios/nrpe.d/commands.cfg is not owned by any package # rpm -Uvv nagios-nrpe-2.15-4.1.i686.rpm D: pool fd: created size 212 limit -1 flags 0 D: pool iob:created size 24 limit -1 flags 0 D: pool mire: created size 88 limit -1 flags 0 D: pool lua:created size 36 limit -1 flags 0 D: pool ts: created size 724 limit -1 flags 0 D: pool gi: created size 96 limit -1 flags 0 D: pool dig:created size 248 limit -1 flags 0 D: pool h: created size 220 limit -1 flags 0 D: Expected size:42149 = lead(96)+sigs(920)+pad(0)+data(41133) D: Actual size:42149 D: pool ds: created size 128 limit -1 flags 0 D: pool tsi:created size 28 limit -1 flags 0 D: pool te: created size 224 limit -1 flags 0 D: pool fi: created size 320 limit -1 flags 0 D: pool al: created size 44 limit -1 flags 0 D: pool bf: created size 28 limit -1 flags 0 D: pool db: created size 188 limit -1 flags 0 D: pool dbi:created size 312 limit -1 flags 0 D: rpmdb: cpus 1 physmem 531Mb D: opening db environment /var/lib/rpm/Packages thread:lock:log:mpool:txn D: opening db index /var/lib/rpm/Packages thread:rdonly:auto_commit mode=0x0 D: opening db index /var/lib/rpm/Name thread:rdonly:auto_commit mode=0x0 D: pool mi: created size 92 limit -1 flags 0 D:upgrade erases nagios-nrpe-2.15-1.i686 D: opening db index /var/lib/rpm/Providename thread:rdonly:auto_commit mode=0x0 D: == +++ nagios-nrpe-2.15-4.1.i686 i686/linux 0x1 D: pool ps: created size 24 limit -1 flags 0 D: opening db index /var/lib/rpm/Filepaths thread:rdonly:auto_commit mode=0x0 D: Requires: /bin/sh YES (db files) D: Requires: /bin/sh YES (db files) D: Requires: /bin/sh YES (db files) D: Requires: /bin/sh YES (db files) D: Requires: /sbin/chkconfig YES (db files) D: Requires: libc.so.6 YES (db provides) D: Requires: libc.so.6(GLIBC_2.0) YES (db provides) D: Requires: libc.so.6(GLIBC_2.1) YES (db provides) D: Requires: libc.so.6(GLIBC_2.15) YES (db provides) D: Requires: libc.so.6(GLIBC_2.3) YES (db provides) D: Requires: libc.so.6(GLIBC_2.3.4)YES (db provides) D: Requires: libc.so.6(GLIBC_2.4) YES (db provides) D: Requires: libcrypto.so.1.0.0YES (db provides) D: Requires: libssl.so.1.0.0 YES (db provides) D: Requires: libwrap.so.0 YES (db provides) D: Requires: nagios-common YES (db provides) D: Requires: rc-scriptsYES (db provides) D: Requires: rtld(GNU_HASH)YES (db provides) D: Dirs: /etc/nagios YES (db files) D: Dirs: /etc/nagios/nrpe.dYES (added files) D: Dirs: /etc/rc.d/init.d YES (db files) D: Dirs: /usr/lib/tmpfiles.d YES (db files) D: Dirs: /usr/sbin YES (db files) D: Dirs: /usr/share/docYES (db files) D: Dirs: /usr/share/doc/nagios-nrpe-2.15 YES (added files) D: Dirs: /var/run YES (db files) D: opening db index /var/lib/rpm/Conflictname thread:rdonly:auto_commit mode=0x0 D: == --- nagios-nrpe-2.15-1.i686 i686/linux 0x1 D: opening db index /var/lib/rpm/Requirename thread:rdonly:auto_commit mode=0x0 D: == recording tsort relations D: Provides: nagios-nrpe = 0:2.15-4.1 YES (added provide) D: ignore package name relation(s) [0] QtGui -> Qt3Support D: ignore package name relation(s) [1] QtGui -> QtSvg D: ignore package name relation(s) [2] ZendFramework-Zend_Cache -> ZendFramework-Zend_Json D: ignore package name relation(s) [3] ZendFramework-Zend_Cache -> ZendFramework-Zend_Log D: ignore package name relation(s) [4] ZendFramework-Zend_Controller -> ZendFramework-Zend_Dojo D: ignore package name relation(s) [5] ZendFramework-Zend_Controller -> ZendFra
Re: rpm overwriting config files again
On May 25, 2015, at 9:01 AM, Elan Ruusamäe wrote: > again rpm is ovewriting existing files, not creating them as .rpmnew when the > file is "new" in the package itself. > Claiming "overwriting" is premature (see below). > imho this got solved at least in 4.5... > Comparisons to rpm-4.x behaviors aren't useful because 1) rpm.org has different (and changed at rpm.org) semantics for %config handling 2) rpm-4.5 went end-of-life back in 2007 and has no upgrade path. And in this case, I believe your claim is incorrect. The code in rpmfiDecideFile() is unchanged (except in some API cosmetic ways afaict) since 09-Jul-2003 when first checked in. Even the API cosmetic changes date back to 2008. So if this "worked" in rpm-4.5, PLD likely patched a different behavior. Not my problem, mon. My belief is as valid as yours until someone (not me) actually provides evidence to the contrary. > scenario: > 1. foo.conf is present in filesystem, but not in package v1.0 > 2. package v1.1 contains foo.conf as %config(noreplace) %verify(not md5 mtime > size) > 3. upgrade package v1.0 -> v1.1 ovewrites foo.conf, not creating > foo.conf.rpmnew > The confusion here starts with "noreplace", which does *NOT* mean Never replace this file. There are several circumstances under which a file marked %config(noreplace) *WILL* be overwritten rather then installed with ".rpmnew", including UNMODIFIED: the ondisk digest is identical to the file digest in the installed package. FILETYPE changes: the file in the installed package has a different type than in the new package > rpm-5.4.15-6.x86_64 > > > log from such system: > > # md5sum /etc/nagios/nrpe.d/commands.cfg > a57588d1e0fb022e0e0bd5945d4ddeb1 /etc/nagios/nrpe.d/commands.cfg > > # rpm -qf /etc/nagios/nrpe.d/commands.cfg > file /etc/nagios/nrpe.d/commands.cfg is not owned by any package > > Executing rpm --upgrade -vh --root / --define _check_dirname_deps 1... > Preparing... ### [100%] > 1:nagios-nrpewarning: /etc/nagios/nrpe.cfg created as > /etc/nagios/nrpe.cfg.rpmnew > ### [100%] > > # rpm -qfc /etc/nagios/nrpe.d/commands.cfg > /etc/nagios/nrpe.cfg > /etc/nagios/nrpe.d/commands.cfg > > # md5sum /etc/nagios/nrpe.d/commands.cfg > 2e4e27f09f7e1efa04c121c6658536a1 /etc/nagios/nrpe.d/commands.cfg > > # ls -l /etc/nagios/nrpe.d/commands.cfg* > -rw-r- 1 root nagios 1.8K May 25 15:43 /etc/nagios/nrpe.d/commands.cfg > If you run with -Uvv (instead of -Uvh), you will see what RPM does with EVERY file in the output. My guess (similar to your previous claim of "overwritten") is that package order ended up as erase-before-install as part of a multiple package upgrade. Was this behavior seen in a multiple package upgrade that you have snipped out? 73 de Jeff ___ pld-devel-en mailing list pld-devel-en@lists.pld-linux.org http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
