mergemaster RFC (long)

[Doug Barton]

Greetings campers,

Mergemaster first made its appearance as a port two years ago tomorrow,
and became part of the FreeBSD source tree last October 20th. Lots of
things have changed since the original version of the program (or
script, whatever), and there are some proposals on the board for it that
I'd really like to get some input on.

For those that aren't aware, mm started its life kind of on a dare. My
life before getting the full time job that I have now was mostly doing
consulting work, so I did a lot of updates remotely. Making the system
safe for remote upgrading has always been a major goal of mine. The
proto-mergemaster was just a little script that I used because I got
tired of forgetting the little bits and pieces. During a conversation on
line about how the configuration system should be expanded, I mentioned
that I had a little script that helped during upgrades, and a lot of
people wanted a copy. Since I had to clean the code up anyway, and since
certain people said it couldn't be done, I created what turned into the
mergemaster we know and love today.

I had some design goals when I started the project, most of which came
from my background in doing remote upgrades. The first goal was to make
mm as secure as possible. There are numerous precautions built in to
avoid overwriting files inappropriately, avoid typical /tmp file
overwriting exploits, etc. I also wanted it to be as independent as
possible from having to know anything about specific files. This would
allow it to continue to be useful no matter what got changed or
upgraded. To accomplish this goal I mercilessly cribbed some ideas from
Nik Clayton's make world page. The next goal was to make the program as
simple as possible given the complexity of some of the stuff that it
deals with. I tried to accomplish that by making it as unix like as
possible, with tools, options, etc. that act as much like "regular" unix
tools as my knowledge and ability allowed. Finally, one of the central
design goals was to never take any action that the user didn't
specifically request. ALL of the defaults in the program are to do
nothing, because in almost all cases that's the safest thing to (not)
do. 

That final goal has been compromised somewhat in the past year, as
different people have requested various hacks to avoid having to look at
certain files that they never want to upgrade. I'm uncomfortable with
this for several reasons. First, because of the way mm works it's
already possible (and in fact, desirable) to "hide" local modifications
to files by keeping their cvs $FreeBSD ($Id) tags the same as the
current version in source. In this way, you only have to deal with the
file when there are actual updates, which I feel strongly is something
you _should_ do anyway. However, over time I've come to accept the fact
that mm now belongs more to the community than to me exclusively, so
what *I* think you should do is less important than it was. 

The other struggle I'm having currently is that the thing is really
becoming a beast. I wanted it to be small, simple and fast, and in the
tradition of unix tools to do one thing and do it well. The thing
already has WAY more options than I ever use, which isn't necessarily
the ultimate barometer, but it is starting to give me the willies. The
other reason I'm growing increasingly concerned is the number of posts
in -questions, et al that go something like; "I see in UPDATING that I
should be using mergemaster, but I tried it and I don't really
understand what the hell it does, so I just didn't bother." I tried to
make the man page as simple as possible, and include stuff that I wish
was present in other man pages, but it's not uncommon for people to not
even bother reading the man page, or read it, get overwhelmed by the
options, give up, and go anyway. 

All of that is background to aid in understanding where I'm coming from
when I talk about the following. There are a couple of categories of
proposed changes. Some are things that I'm definitely going to do. Some
are things that I plan to do, and am currently working on and/or have
patches for. I'm including those two categories because if I don't
someone will feel the need to suggest them. :)  The final category are
things that I would like comments about, hence the subject of this
e-mail. 

First, the things I am definitely going to do. Christian "naddy"
Weisgerber has taken on the task of porting mm to openbsd. He has made
some very reasonable requests that will make his life easier and reduce
gratuitous differences between versions. Also, several people have asked
for
the ability to specify DESTDIR, which is an easy fix and definitely an
understandable request. 

Second, the things I am planning to do. Several people have asked for
mergemaster to make use of cvs, going all the way back to freebsdcon.
However, it's only been recently that anyone has given me actual
concrete plans on things

Re: mergemaster RFC (long)

[Brian Somers]

[.]
>   First, the things I am definitely going to do. Christian "naddy"
> Weisgerber has taken on the task of porting mm to openbsd. He has made
> some very reasonable requests that will make his life easier and reduce
> gratuitous differences between versions. Also, several people have asked
> for
> the ability to specify DESTDIR, which is an easy fix and definitely an
> understandable request. 

I think it would be nice to aim to keep the two scripts exactly the 
same, using `uname` when it's really necessary.  IMHO this really 
helps when maintaining code - if there are different versions, the 
tendency is to update the one you use most, and ``mean to get around 
to'' updating the other(s) later.  Of course the ``later'' tends to 
stop happening after a while...

[.]
> script. I'm going to add variables to the .mergemasterrc file for
> mergemaster-start and mergemaster-end scripts (names may change). These
> scripts will run after the temproot environment is built, and after mm
> is done with it's run. These scripts would be source'd from within mm so
> all the variables will be available to them.  This should allow people
> to add in their own "options" that I don't want to add to the program. 

I like this idea.  Perhaps the man page should have a HOWTO section 
that describes how to ``not compare files'' by deleting them in the 
start script etc.

> This is cobbled together from various people's suggestions.
> We need a list of files somewhere (probably in .mergemasterrc) that
> people don't ever want to have to deal with via the "normal" mm process.
> If a file on that list has been updated (according to the cvs tag) then
> instead of presenting that file to the user, it will be ignored the
> first time through. Then, at the end of the script there will be a
> printout similar to what happens when the user leaves a file to do by
> hand later. Something like:

I think having

  IGNOREFILES="a b c"

isn't necessary when it's as easy to have

  rm a b c

in your start script.  It seems like overkill to handle ignored files 
specifically.

>   Next on the list of things I'm open to suggestions on is a small script
> I wrote that compares the variables in /etc/rc.conf[.local] to the ones
> in /etc/defaults/rc.conf, and does the same for make.conf.
[.]

I don't really think it matters where this happens in mergemaster, 
but it would be nice to include periodic.conf too.  Of course you 
can't generically include everything in defaults/ as pccard.conf is 
probably something to avoid.

>   This also brings up the question of which files in /etc are good to
> examine before a world build.
[.]

Yes, too much handholding I think.

>   There was a PR filed in regards to the current situation with the new
> sendmail moving the aliases file to /etc/mail, and mergemaster's
[.]

Yep, this is definitely an SEP !

>   I have an idea related to getting users to actually READ the man page
[.]

Yeuch ! :-)

>   And the last of the ideas I want feedback on is related to the problem
> of determining permissions on the files to be installed. To my knowledge
[.]

Perhaps the best thing here is to make mtree capable of understanding 
rwxrwxrwx type modes, and then generate an mtree spec file based on 
the stuff in temproot and apply it to destdir ?

> Doug
> -- 
> "The dead cannot be seduced."
>   - Kai, "Lexx"
> 
>   Do YOU Yahoo!?

-- 
Brian <[EMAIL PROTECTED]>
     
Don't _EVER_ lose your sense of humour !




To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message

Re: mergemaster RFC (long)

[Aleksandr A.Babaylov]

Doug Barton writes:
..
>   And the last of the ideas I want feedback on is related to the problem
> of determining permissions on the files to be installed. To my knowledge
> there is no existing C utility that will tell you the octal permissions
ports/sysutils/stat, for example

0cicuta~(12)>stat -f'%p' .
0755/drwxr-xr-x

But I have stat patched to delete  File: \"%s\" clause from
printing if no need for it.
It is short program and useful in base system

0cicuta~(13)>ls -al `which stat`
 6 -r-xr-xr-x  1 root  wheel  5720 26 Á×Ç 00:58 /usr/local/bin/stat*

> on a file, and preferably also combine them with the umask to spit out
> what the permissions should be. My current solution for that is a couple
> of one line perl scripts, but I really want to eliminate that as my
> contribution to the "don't make anything in the base depend on perl"
> campaign. I have written a little tiny program in C to do this which
> could be included along with mergemaster, what do people think? It is
> possible to do the same thing in sh, but it's very cumbersome, unless
> someone can suggest some combination of existing utilities that I
> haven't considered yet. The main problem with doing it in sh (which I've
> done in the past) is that the umask effectively means "different" things
> if it's a binary/directory or if it's a regular file, so even though
> getting the permissions is easy (with ls, whatever) combining it with
> the umask is hard. Of course, I could always take that whole section
> out, and just install the files with the permissions created by
> /usr/src/etc/Makefile. I see less need now of custom permissions than I
> did ages ago, but what do others think?
..

-- 
@BABOLO  http://links.ru/


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message

Re: mergemaster RFC (long)

[Doug Barton]

"Aleksandr A.Babaylov" wrote:
> 
> Doug Barton writes:
> ..
> >   And the last of the ideas I want feedback on is related to the problem
> > of determining permissions on the files to be installed. To my knowledge
> > there is no existing C utility that will tell you the octal permissions

> ports/sysutils/stat, for example

Thank you for the reference. I should have stated more explicitly that
it needs to be part of the base already. You're correct in stating that
a program to manipulate the stat(2) function is easy to write, even I
can write one. :)

Doug
-- 
"The dead cannot be seduced."
- Kai, "Lexx"

Do YOU Yahoo!?


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message

Re: mergemaster RFC (long)

[Christian Weisgerber]

Brian Somers <[EMAIL PROTECTED]> wrote:

> > First, the things I am definitely going to do. Christian "naddy"
> > Weisgerber has taken on the task of porting mm to openbsd.
> 
> I think it would be nice to aim to keep the two scripts exactly the 
> same, using `uname` when it's really necessary.

If I have interpreted the noises Theo has made correctly, he wants
mergemaster in the base tree.  I don't think he'll keep a "case
`uname` ..." in there.

Most of the diff deals with two simple differences:
- mergemaster uses "read -p " throughout.  That fails for
  OpenBSD's /bin/sh (pdksh), where "read -p" means something entirely
  different.
- On OpenBSD, "install" is synonymous to "install -c".  FreeBSD still
  has the old behavior where plain "install" deletes the source file.

If we can get rid of those, the actual differences become more
visible.

Oh, and changing every instance of "FreeBSD" into "${OPSYS}" or
some such would remove another few diff lines.

> I think having
> 
>   IGNOREFILES="a b c"
> 
> isn't necessary when it's as easy to have
> 
>   rm a b c
> 
> in your start script.  It seems like overkill to handle ignored files 
> specifically.

Well, I don't know.  Something like "IGNOREFILES=..." was the first
thing to come to my mind.

-- 
Christian "naddy" Weisgerber  [EMAIL PROTECTED]



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message

Re: mergemaster RFC (long)

[Brian Somers]

> Brian Somers <[EMAIL PROTECTED]> wrote:
> 
> > >   First, the things I am definitely going to do. Christian "naddy"
> > > Weisgerber has taken on the task of porting mm to openbsd.
> > 
> > I think it would be nice to aim to keep the two scripts exactly the 
> > same, using `uname` when it's really necessary.
> 
> If I have interpreted the noises Theo has made correctly, he wants
> mergemaster in the base tree.  I don't think he'll keep a "case
> `uname` ..." in there.

Of course if we didn't need it... that'd be far better :-)

> Most of the diff deals with two simple differences:
> - mergemaster uses "read -p " throughout.  That fails for
>   OpenBSD's /bin/sh (pdksh), where "read -p" means something entirely
>   different.

That's not too bad as the read -p can be changed to echo/read.

> - On OpenBSD, "install" is synonymous to "install -c".  FreeBSD still
>   has the old behavior where plain "install" deletes the source file.

Maybe the best bet here is to have ``INSTALL=install'' at the top of 
the script (``INSTALL="install -c" for FreeBSD).  It's a difference, 
but managable.

> If we can get rid of those, the actual differences become more
> visible.
> 
> Oh, and changing every instance of "FreeBSD" into "${OPSYS}" or
> some such would remove another few diff lines.

Aye.  Looks like we're thinking along the same lines.

I'll have to talk to Wes at BSDCon about trying to get the NetBSD 
guys to introduce it there too - maybe some of the NetBSD developers 
will be there too.

> -- 
> Christian "naddy" Weisgerber  [EMAIL PROTECTED]

-- 
Brian <[EMAIL PROTECTED]>
     
Don't _EVER_ lose your sense of humour !




To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message

Re: mergemaster RFC (long)

[Josef Karthauser]

Another thing that would be very useful is that during a
merge of two files that it's possible to specify both
the left hand side and the right hand side.  That would
fix cases like:

Orig:

# $FreeBSD: src/etc/shells,v 1.4 SOMEDATE$
#
# List of acceptable shells for chpass(1).
# Ftpd will not allow users to connect who are not using
# one of these shells.

/bin/sh
/bin/csh
/usr/local/bin/zsh


The new 1.5 version has: /bin/tcsh in it.  Mergemaster won't let
me have both my /usr/local/bin/zsh and the /bin/tcsh.  During merge
I can have either but not both :(

Joe


annoying when you onl
On Wed, Sep 20, 2000 at 09:34:03PM +0100, Brian Somers wrote:
> > Brian Somers <[EMAIL PROTECTED]> wrote:
> > 
> > > > First, the things I am definitely going to do. Christian "naddy"
> > > > Weisgerber has taken on the task of porting mm to openbsd.
> > > 
> > > I think it would be nice to aim to keep the two scripts exactly the 
> > > same, using `uname` when it's really necessary.
> > 
> > If I have interpreted the noises Theo has made correctly, he wants
> > mergemaster in the base tree.  I don't think he'll keep a "case
> > `uname` ..." in there.
> 
> Of course if we didn't need it... that'd be far better :-)
> 
> > Most of the diff deals with two simple differences:
> > - mergemaster uses "read -p " throughout.  That fails for
> >   OpenBSD's /bin/sh (pdksh), where "read -p" means something entirely
> >   different.
> 
> That's not too bad as the read -p can be changed to echo/read.
> 
> > - On OpenBSD, "install" is synonymous to "install -c".  FreeBSD still
> >   has the old behavior where plain "install" deletes the source file.
> 
> Maybe the best bet here is to have ``INSTALL=install'' at the top of 
> the script (``INSTALL="install -c" for FreeBSD).  It's a difference, 
> but managable.
> 
> > If we can get rid of those, the actual differences become more
> > visible.
> > 
> > Oh, and changing every instance of "FreeBSD" into "${OPSYS}" or
> > some such would remove another few diff lines.
> 
> Aye.  Looks like we're thinking along the same lines.
> 
> I'll have to talk to Wes at BSDCon about trying to get the NetBSD 
> guys to introduce it there too - maybe some of the NetBSD developers 
> will be there too.
> 
> > -- 
> > Christian "naddy" Weisgerber  [EMAIL PROTECTED]
> 
> -- 
> Brian <[EMAIL PROTECTED]>
>      
> Don't _EVER_ lose your sense of humour !
> 
> 
> 
> 
> To Unsubscribe: send mail to [EMAIL PROTECTED]
> with "unsubscribe freebsd-hackers" in the body of the message

-- 
Josef KarthauserFreeBSD: How many times have you booted today?
Technical Manager   Viagra for your server (http://www.uk.freebsd.org)
Pavilion Internet plc.  [[EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED]]


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message

RE: mergemaster RFC (long)

[Peter Avalos]

%?
l:  use the left version
r:  use the right version
e l:edit then use the left version
e r:edit then use the right version
e b:edit then use the left and right versions concatenated
e:  edit a new version
s:  silently include common lines
v:  verbosely include common lines
q:  quit

Look like 'e b' was designed for that.


Peter Avalos
TheShell.com

-BEGIN GEEK CODE BLOCK-
Version: 3.12
GCS/ED/B d-(+) s:+> a-- C++$ UBLO$ P+ L E- W+ N+ o? K? w(++) !O M-
V- PS+ PE++ Y+ PGP++ t+@ 5 X- R- tv+ b++ DI- D-- G e>+++ h-- r++ y++
--END GEEK CODE BLOCK--

> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED]]On Behalf Of Josef Karthauser
> Sent: Thursday, September 21, 2000 12:45 PM
> To: Brian Somers
> Cc: Christian Weisgerber; [EMAIL PROTECTED]
> Subject: Re: mergemaster RFC (long)
>
>
>
> Another thing that would be very useful is that during a
> merge of two files that it's possible to specify both
> the left hand side and the right hand side.  That would
> fix cases like:
>
> Orig:
>
> # $FreeBSD: src/etc/shells,v 1.4 SOMEDATE$
> #
> # List of acceptable shells for chpass(1).
> # Ftpd will not allow users to connect who are not using
> # one of these shells.
>
> /bin/sh
> /bin/csh
> /usr/local/bin/zsh
>
>
> The new 1.5 version has: /bin/tcsh in it.  Mergemaster won't let
> me have both my /usr/local/bin/zsh and the /bin/tcsh.  During merge
> I can have either but not both :(
>
> Joe
>
>
> annoying when you onl
> On Wed, Sep 20, 2000 at 09:34:03PM +0100, Brian Somers wrote:
> > > Brian Somers <[EMAIL PROTECTED]> wrote:
> > >
> > > > >   First, the things I am definitely going to do.
> Christian "naddy"
> > > > > Weisgerber has taken on the task of porting mm to openbsd.
> > > >
> > > > I think it would be nice to aim to keep the two scripts exactly the
> > > > same, using `uname` when it's really necessary.
> > >
> > > If I have interpreted the noises Theo has made correctly, he wants
> > > mergemaster in the base tree.  I don't think he'll keep a "case
> > > `uname` ..." in there.
> >
> > Of course if we didn't need it... that'd be far better :-)
> >
> > > Most of the diff deals with two simple differences:
> > > - mergemaster uses "read -p " throughout.  That fails for
> > >   OpenBSD's /bin/sh (pdksh), where "read -p" means something entirely
> > >   different.
> >
> > That's not too bad as the read -p can be changed to echo/read.
> >
> > > - On OpenBSD, "install" is synonymous to "install -c".  FreeBSD still
> > >   has the old behavior where plain "install" deletes the source file.
> >
> > Maybe the best bet here is to have ``INSTALL=install'' at the top of
> > the script (``INSTALL="install -c" for FreeBSD).  It's a difference,
> > but managable.
> >
> > > If we can get rid of those, the actual differences become more
> > > visible.
> > >
> > > Oh, and changing every instance of "FreeBSD" into "${OPSYS}" or
> > > some such would remove another few diff lines.
> >
> > Aye.  Looks like we're thinking along the same lines.
> >
> > I'll have to talk to Wes at BSDCon about trying to get the NetBSD
> > guys to introduce it there too - maybe some of the NetBSD developers
> > will be there too.
> >
> > > --
> > > Christian "naddy" Weisgerber
> [EMAIL PROTECTED]
> >
> > --
> > Brian <[EMAIL PROTECTED]>
> 
> >   <http://www.Awfulhak.org>
> 
> > Don't _EVER_ lose your sense of humour !
> >
> >
> >
> >
> > To Unsubscribe: send mail to [EMAIL PROTECTED]
> > with "unsubscribe freebsd-hackers" in the body of the message
>
> --
> Josef Karthauser  FreeBSD: How many times have you booted today?
> Technical Manager Viagra for your server (http://www.uk.freebsd.org)
> Pavilion Internet plc.  [[EMAIL PROTECTED], [EMAIL PROTECTED],
> [EMAIL PROTECTED]]
>
>
> To Unsubscribe: send mail to [EMAIL PROTECTED]
> with "unsubscribe freebsd-hackers" in the body of the message



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message

Re: mergemaster RFC (long)

[Brian Somers]

> 
> Another thing that would be very useful is that during a
> merge of two files that it's possible to specify both
> the left hand side and the right hand side.  That would
> fix cases like:
> 
> Orig:
> 
> # $FreeBSD: src/etc/shells,v 1.4 SOMEDATE$
> #
> # List of acceptable shells for chpass(1).
> # Ftpd will not allow users to connect who are not using
> # one of these shells.
> 
> /bin/sh
> /bin/csh
> /usr/local/bin/zsh
> 
> 
> The new 1.5 version has: /bin/tcsh in it.  Mergemaster won't let
> me have both my /usr/local/bin/zsh and the /bin/tcsh.  During merge
> I can have either but not both :(
> 
> Joe

You can ``e b'' (edit both) from the sdiff prompt.
-- 
Brian <[EMAIL PROTECTED]>
     
Don't _EVER_ lose your sense of humour !




To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message