Re: Should I be able to use mergemaster with freebsd-update?
Hi all, A small followup: Looks like freebsd-update does try to rebuild the password database but does not quite succeed, leaving binary files in somewhat corrupted state, this leading to some problems when trying to add new users later. This is discussed here: http://www.freebsd.org/cgi/query-pr.cgi?pr=180241 However the master.passwd is fine, thus as a workaround you can simply run vipw to resave master.passwd, then vipw regenerates binary password databases correctly and everything works quite nicely. Best wishes Eugene -Original Message- From: Eugene Sent: Tuesday, July 02, 2013 8:37 PM To: freebsd-questions@freebsd.org Subject: Re: Should I be able to use mergemaster with freebsd-update? Hi all, In case anybody was following this discussion, I have successfully upgraded the system from 8.2 to 8.4 using freebsd-update. The process did have some glitches (in retrospect, minor ones) but mostly they were not related to freebsd-update (like some issues with gmirror and firewall configurations). The data merging phase was quite bearable and reasonable (if a bit tedious) and all the databases got properly updated. Thanks to everyone involved! Eugene -Original Message- From: Mike Brown Sent: Wednesday, June 26, 2013 6:22 AM To: freebsd-questions@freebsd.org Subject: Re: Should I be able to use mergemaster with freebsd-update? On Tue, Jun 25, 2013, at 15:29, Eugene wrote: I do not quite understand. Is the freebsd-update upgrade process completely broken? IMHO it is partially broken; I'm not doing anything special. How broken it is depends on what's getting changed. Most of what the system is designed to do, it indeed does very well. It also overlaps some of the functionality of mergemaster in that it automatically merges as many files as it can, which is nice. Where it is under-designed and under-implemented is in its rudimentary handling of un-mergeable files, and in its total lack of support for the regeneration of /etc/*.db files (like the, uh, rather important password database) and sendmail aliases - things that you would handle via mergemaster in an ordinary, source-based upgrade, but which you must now figure out how to do by hand, without any guidance, and they really don't make it easy for you. <...> ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org" ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Should I be able to use mergemaster with freebsd-update?
Hi all, In case anybody was following this discussion, I have successfully upgraded the system from 8.2 to 8.4 using freebsd-update. The process did have some glitches (in retrospect, minor ones) but mostly they were not related to freebsd-update (like some issues with gmirror and firewall configurations). The data merging phase was quite bearable and reasonable (if a bit tedious) and all the databases got properly updated. Thanks to everyone involved! Eugene -Original Message- From: Mike Brown Sent: Wednesday, June 26, 2013 6:22 AM To: freebsd-questions@freebsd.org Subject: Re: Should I be able to use mergemaster with freebsd-update? On Tue, Jun 25, 2013, at 15:29, Eugene wrote: I do not quite understand. Is the freebsd-update upgrade process completely broken? IMHO it is partially broken; I'm not doing anything special. How broken it is depends on what's getting changed. Most of what the system is designed to do, it indeed does very well. It also overlaps some of the functionality of mergemaster in that it automatically merges as many files as it can, which is nice. Where it is under-designed and under-implemented is in its rudimentary handling of un-mergeable files, and in its total lack of support for the regeneration of /etc/*.db files (like the, uh, rather important password database) and sendmail aliases - things that you would handle via mergemaster in an ordinary, source-based upgrade, but which you must now figure out how to do by hand, without any guidance, and they really don't make it easy for you. <...> ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Should I be able to use mergemaster with freebsd-update?
On Wed, Jun 26, 2013, at 2:07, Mike Brown wrote: > > Next step, I think, is reboot, before another 'freebsd-update install' > run. > I'm worried something is still amiss, though, so I'm holding off for now. > :( When in doubt: fetch source, build, install, and use mergemaster. Then reboot. Better safe than sorry. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Should I be able to use mergemaster with freebsd-update?
I wrote: > The main problem this time is that I'm not so lucky with the password files, > because for 8.4, freebsd-update has fetched new, stock .db files to put in > /etc. Whoa, sorry, I misspoke here. freebsd-update asked me, after the merges, to approve unspecified differences in pwd.db and spwd.db. I assumed that it had fetched those files as part of the 8.4 distribution. But http://svnweb.freebsd.org/base/release/8.4.0/etc/ seems to indicate that's not what happened; only master.passwd was changed. I'm looking through the freebsd-update code now. I see it does actually do some special handling of master.passwd, but not until you do your 'freebsd-update install'. At that point, it will look at /etc/master.passwd and see if it's newer than /etc/pwd.db or /etc/spwd.db, and it will run pwd_mkdb. It doesn't use the -p flag, so I guess it doesn't care about passwd. This pwd_mkdb run didn't happen for me, though, since my 'freebsd-update install' run didn't actually put the new master.passwd file, or anything else, into /etc yet. I thought it would, but I don't understand it, really. So I don't see how it's supposed to work. To summarize: 1. I did the initial 'freebsd-update -r 8.4-RELEASE upgrade' 2. When prompted, I did all the merges it needed me to do by hand. 3. When prompted, I approved all the diffs. Two of the diffs were unspecified pwd.db & spwd.db changes, which caused me some alarm. 4. I looked in the staging area and found that these were empty files. 5. I looked in /etc and nothing new had been placed there yet. 6. I did the 'freebsd-update install' and checked /etc again; still nothing. 7. Afraid of rebooting with bogus password database files staged, I generated proper pwd.db, spwd.db, and passwd files myself, and put them in the staging area. Next step, I think, is reboot, before another 'freebsd-update install' run. I'm worried something is still amiss, though, so I'm holding off for now. :( ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Should I be able to use mergemaster with freebsd-update?
On Tue, Jun 25, 2013, at 15:29, Eugene wrote: > I do not quite understand. Is the freebsd-update upgrade process > completely broken? IMHO it is partially broken; I'm not doing anything special. How broken it is depends on what's getting changed. Most of what the system is designed to do, it indeed does very well. It also overlaps some of the functionality of mergemaster in that it automatically merges as many files as it can, which is nice. Where it is under-designed and under-implemented is in its rudimentary handling of un-mergeable files, and in its total lack of support for the regeneration of /etc/*.db files (like the, uh, rather important password database) and sendmail aliases - things that you would handle via mergemaster in an ordinary, source-based upgrade, but which you must now figure out how to do by hand, without any guidance, and they really don't make it easy for you. When I upgraded from 8.1 to 8.3, I avoided the issue altogether by not really merging anything; when dumped into the empty text editor, I just loaded my old files and made no changes. In the Handbook, there's an assumption that people who do this will go back later and figure out what merges are needed, but the resources you need to do that don't exist; if you don't do the merge when prompted, you don't get a second chance. In fact, even if you do it when prompted, you need to get it right, or start the whole process over. My upgrade to 8.3 worked out OK because I got lucky; freebsd-update hadn't fetched new, stock password database files. The unmergeable files were all text files, nothing requiring anything to be regenerated. But this time around, for 8.3 to 8.4, I am trying to do everything I'm supposed to, actually merging when prompted. The fact that it's a *really* manual process is a pain, but as I mentioned, I found a way to at least run sdiff from another window, which made it a lot easier, although still more tedious than it should be. The main problem this time is that I'm not so lucky with the password files, because for 8.4, freebsd-update has fetched new, stock .db files to put in /etc. So, yes, I was able to merge master.passwd & passwd, but that's not very helpful since the .db files won't be in sync with them. If allow my custom password database to be overwritten with these new, stock .db files, obviously that's bad. And because freebsd-update makes no special allowance for the .db files, it actually put a zero-byte file in the staging area instead of the real .db file (as if it were going to have me modify it with an editor). So if I proceed, my password database will actually be overwritten with an empty file, which I believe would be a disaster. The solution, I feel, is to: 1. make freebsd-update recognize files that most likely need to be regenerated instead of replaced - /etc/*.db, at least, if not also any other binary file, and some of the things that would be generated by 'make' in /etc/mail. The user should be informed that these files need to be regenerated, if there's no way to just regenerate them automatically when their companion source files have been updated or merged. 2. make freebsd-update run mergemaster on the unmergeable text files, instead of dumping the user into an empty text editor for each one. For each file that can't be automatically merged, mergemaster will give the user the opportunity to choose whether to keep the old file, replace it with the new file, interactively merge them via sdiff, or do nothing. It is also smart enough to realize that when certain files are being touched, such as /etc/master.passwd, /etc/mail/aliases, etc. you'll need to run pwd_mkdb, cap_mkdb, services_mkdb, or newaliases...and it will run those for you (or remind you to do it). For this to work, mergemaster would need some tweaking to deal with freebsd-update's staging area, and to not duplicate any of the work that freebsd-update does. I keep hoping that maybe there's some nuance of the process that I'm missing, and that all of this really is not a problem.. user error, or not reading the docs carefully enough, you know? But Mark Felder's comments seem to confirm that it's a real issue. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Should I be able to use mergemaster with freebsd-update?
On Tue, Jun 25, 2013, at 15:29, Eugene wrote: > Hi all, > > I do not quite understand. Is the freebsd-update upgrade process > completely > broken? Or is it some special mode? Or was it broken recently? > Because some time ago I have upgraded from 8.1 to 8.2 quite nicely, with > editor-based merging of config files, and was planning to upgrade to 8.4 > soon (especially as 8.2 is already not compatible with some ports). > It depends on how many changes happen between the releases. Have you tried taking 7.x to 9.x before? You'll have to deal with that editor for merging many, many files. Maybe nearly everything in /etc. It's quite time consuming, whereas I can get mergemaster to auto-merge all of those files and only show me the 5 that I've personally touched. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Should I be able to use mergemaster with freebsd-update?
Hi all, I do not quite understand. Is the freebsd-update upgrade process completely broken? Or is it some special mode? Or was it broken recently? Because some time ago I have upgraded from 8.1 to 8.2 quite nicely, with editor-based merging of config files, and was planning to upgrade to 8.4 soon (especially as 8.2 is already not compatible with some ports). Best wishes Eugene -Original Message- From: Mike Brown Sent: Tuesday, June 25, 2013 12:14 PM To: freebsd-questions@freebsd.org Subject: Re: Should I be able to use mergemaster with freebsd-update? I'm using freebsd-update to upgrade my system to the latest minor version (from 8.3-RELEASE to 8.4-RELEASE). I'm surprised that the merge handling system isn't more robust. When upgrading the old way, from source, I was used to using mergemaster to handle any merges that couldn't be done automatically. But when using freebsd-update, it seems that any failed merges require that you get dumped into an empty text editor for each file. [...] As I continue with this process, doing all the mergemaster tasks manually, I'm finding that the situation is even worse than I first realized. <> ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Should I be able to use mergemaster with freebsd-update?
On Tue, Jun 25, 2013, at 3:14, Mike Brown wrote: > > Well, thanks for reading this far. I'm scared to death to reboot now, > since my > server is in another city, but we'll see how it goes. > I always avoid freebsd-update when moving between releases simply because of this atrocity. If it requires we setup a stupid kickstarter to fund a developer to sit down and rip into freebsd-update so it uses mergemaster I would be incredibly thankful. I don't know how anyone can upgrade between FreeBSD releases without an /etc/mergemaster.rc with the following settings: AUTO_INSTALL='yes' AUTO_UPGRADE='yes' # keep our custom motd IGNORE_FILES='/etc/motd' # Do not display changes that only affect whitespace DIFF_FLAG='-Bub' FREEBSD_ID='yes' DELETE_STALE_RC_FILES='yes' ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Should I be able to use mergemaster with freebsd-update?
> I'm using freebsd-update to upgrade my system to the latest minor version
> (from 8.3-RELEASE to 8.4-RELEASE).
>
> I'm surprised that the merge handling system isn't more robust. When
> upgrading
> the old way, from source, I was used to using mergemaster to handle any
> merges that couldn't be done automatically.
>
> But when using freebsd-update, it seems that any failed merges require that
> you get dumped into an empty text editor for each file.
> [...]
As I continue with this process, doing all the mergemaster tasks manually, I'm
finding that the situation is even worse than I first realized.
First, the relatively painless part. As I mentioned, after running
'freebsd-update -r 8.4-RELEASE upgrade', I had to deal with the un-mergeable
files.
Although mergemaster apparently isn't an option, its interactive merge
function is really just a front-end for sdiff, so I found that I could
replicate that part of its functionality by doing this in a separate window
(-w 100 because I use a 100-column terminal):
cd /var/db/freebsd-update/merge/8.4-RELEASE
find -X . -type f | xargs -n 1 -o -I % sh -c '{ echo Now processing %.
left=current, right=new, help="?"; sdiff -d -w 100 -o ../new/% ../old/% %; }'
This populated my 'new' directory with merged files, so that (in the first
window) when I opened the text editor for each one, I only had to just give it
a once-over and exit the editor.
Among the diffs in this 8.3 to 8.4 upgrade were changes to /etc/master.passwd
and /etc/passwd, to add the 'auditdistd' and 'hast' users. As reported in
March 2012 [1] in relation to 8.x to 9.x upgrades, this won't work as
expected, because freebsd-update doesn't run pwd_mkdb after the master.passwd
update.
Now the real hurt begins; in the 8.3 to 8.4 upgrade, it's even worse.
Once I saved all the files in the editor, I was prompted to approve a diff for
each one. I had to answer "y" or the entire process aborts.
Among the changes I was asked to approve, besides visible diffs, were
unspecified differences in /etc/pwd.db and /etc/spwd.db, the binary files that
contain the password database. There's no choice but to answer "y" and
approve them, and I don't get any opportunity to rebuild them properly.
So apparently, freebsd-update wants to install new, stock password databases,
which are out-of-sync with my customized, merged master.passwd & passwd files.
(And because of the way the freebsd-update system works, what I actually
approved were empty, 0-byte files, the result of the failed merges.)
What would happen if I just let it do this? Surely I wouldn't be able to log
in, after the reboot, right?
After approving the files (again, I had no choice!), I was presented with
lists of all the files that would be deleted, added, and modified. Sure
enough, bad /etc/pwd.db and /etc/spwd.db files were in the list.
At this point, the merge folders were now gone; I no longer had the new
master.passwd in a recognizable place. So I thought, OK, I'll run
'freebsd-update install' and hope that the new files end up in /etc. Then I
could just run 'pwd_mkdb -p /etc/master.passwd' to regenerate passwd, pwd.db
and spwd.db before my reboot.
But the 'freebsd-update install' didn't put them there yet; I guess that
doesn't happen till after the reboot. So they're still sitting in a staging
folder, now gzipped and with obfuscated names, indexed in a separate file.
Averting this disaster-in-the-making is not at all straightforward:
cd /var/db/freebsd-update
mkdir -m 0700 /tmp/oldpwdfiles
zcat files/`grep '^/etc/master\.passwd' install.LYQAJQ/INDEX-NEW | cut -d \| -f
7`.gz > /tmp/oldpwdfiles/master.passwd
zcat files/`grep '^/etc/passwd' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz >
/tmp/oldpwdfiles/passwd
zcat files/`grep '^/etc/pwd\.db' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz
> /tmp/oldpwdfiles/pwd.db
zcat files/`grep '^/etc/spwd\.db' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz
> /tmp/oldpwdfiles/spwd.db
mkdir -m 0700 /tmp/newpwdfiles
pwd_mkdb -d /tmp/newpwdfiles -p /tmp/oldpwdfiles/master.passwd
gzip /tmp/newpwdfiles/*
mv /tmp/newpwdfiles/master.passwd.gz files/`grep '^/etc/master\.passwd'
install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz
mv /tmp/newpwdfiles/passwd.gz files/`grep '^/etc/passwd'
install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz
mv /tmp/newpwdfiles/pwd.db.gz files/`grep '^/etc/pwd\.db'
install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz
mv /tmp/newpwdfiles/spwd.db.gz files/`grep '^/etc/spwd\.db'
install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz
rm -fr /tmp/oldpwdfiles /tmp/newpwdfiles
I'm really shocked that it came to this. Did I just overlook the
"--no-surprises" option in freebsd-update?
And now, before I reboot, I have to figure out how to handle the changes that
got made in /etc/mail ... ordinarily I'd run 'make all install restart' in
there. Not an option till after reboot, though. At least it's not crucial for
the reboot to work.
Again, this is something that mergemaster w
