Re: mergemaster improvement (auto-update for not modified files)
Hello! Daniel O'Connor wrote: You don't need to download anything to start using etcmerge, you can just use the files from your last mergemaster. For etcmerge it is need to run mergemaster one last time, or use etc archive for some release. So if mergemaster will be improved, it will be better for etcmerge :) Well, you can use etcmerge if you haven't changed anything in /etc - ie on a fresh install. I made another patch to mergemaster http://people.freebsd.org/~den/scripts/mergemaster/mergemaster-check-store.diff It poorly tested and have known problems, so it is not recommended for now, but shows how simular functionality can be included into mergemaster (with this patch mergemaster can collect some checksums itself). Unfortunately, this will help only on further updates. Even if the checksum test is added to mergemaster it only covers one of the cases etcmerge handles, it still doesn't do a 3 way merge. The merge etcmerge does is very nice for removing changes to files you don't care about. One case is the one important step to the better world :) I hope that we can made some small improvement to mergemaster before 5.5-RELEASE. And it is unlikely that we can introduce something other than mergemaster in RELENG_5. 264k is a pretty large file to commit to the repo.. Yes, I know. And don't sure that it is some need to commit this file. Anyway, this file is less that INDEX, and unlike INDEX, will not rapidly changed. Checksum database will grow slowly. The INDEX file isn't in CVS anymore.. The reasons why it was removed can't be applied to checksums file. It probably won't grow very fast, but IMO it seems like a bit of a kludge. I published my opinion here: http://people.freebsd.org/~den/scripts/mergemaster/checksums-commit.html Maybe it is not bikeshed discussion, but I don't want to continue it in [EMAIL PROTECTED] den@ is appropriate place :) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: mergemaster improvement (auto-update for not modified files)
Hello! There is one major problem here: This can be done in single-user mode only if your have local CVS repository, because if local CVS is not exist, anoncvs is used. Possible solutions: There was some points that I missed: 1. Pre-checkout files in pre-buildworld mode. pre-buildworld is executed in single-user mode too, so it is impossible to do pre-checkout in this mode. 2. Setup CVSup collection for subset of required files (is it possible?), and then keep this small collection up-to-date locally. src-etc cvs collection already here. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: mergemaster improvement (auto-update for not modified files)
Hello! Anton Berezin wrote: On Tue, May 03, 2005 at 01:27:09PM +0400, Denis Peplin wrote: The mergemaster with this is test patch (attached) can auto-update files that was not modified. Please also see http://www.freebsd.org/cgi/getmsg.cgi?fetch=0+0+/usr/local/www/db/text/2005/freebsd-audit/20050116.freebsd-audit Dougb at some point promised to get back with his comments, but it never happened. :-( Thanks, I will look (most likely at the next week). ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: mergemaster improvement (auto-update for not modified files)
Hello! Julian Elischer wrote: Mergemaster could keep checksums of known revisions. it wouldn't take much to have just one file with the last 35 checksums of each file. (maybe with the $FreeBSD$ line removed if it differs..) Just implemented both variants (but second is not by count) 1. Full checksum story for every file (~ 650k). 2. Separated by date checksums (year 2000 was taken for example). New patch to mergemaster: http://people.freebsd.org/~den/scripts/mergemaster/mergemaster-checksum.diff Script to collect checksums: http://people.freebsd.org/~den/scripts/mergemaster/etc_collect_checksums-splitted.sh This script takes few hours to complete, but checksums list available too. Checksums, other links, TODO and full story here: http://people.freebsd.org/~den/scripts/mergemaster/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: mergemaster improvement (auto-update for not modified files)
Hello! Juergen Unger wrote: On Tue, May 03, 2005 at 11:15:51AM -0700, Julian Elischer wrote: files that I have not touched are at default state and I wnat them to move to teh new default state. Files I have touched, I want to look at by hand. so do I. And to go a step further: why do we at all have to run this things in single-user mode ? Not that I am not aware of the technical reasons for it BUT: why don't we run both the installworld and the mergemaster jobs in a manner where they only _preparing_ the installation of the update ? ... maybe creating a jobfile which could be run automatically during reboot to make the update instantaneous happen without a long downtime to do it all manually in single-user mode over a slow serial console ? Ideas like this are definitely flying around. In addition I can say that it will be good to recieve control back even if something goes wrong on such auto-update. Maybe statically-linked sshd in root directory would help in such (and many other) situations. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: mergemaster improvement (auto-update for not modified files)
Hello! Daniel O'Connor wrote: On Fri, 6 May 2005 20:18, Denis Peplin wrote: You know you can just use etcmerge to do this.. It does a 3 way merge between your files and the old and new revisions. The only down side is that it's UI is totally unlike mergemaster so it can be a bit strange to get used to. Thank for pointing me to this script, I will look into it for ideas related to merging files that was modified. But for not modified files etcmerge is too complicated. Updating for not modified files should be done in fully automated mode. And for comparision, this file http://people.freebsd.org/~den/scripts/mergemaster/sums-etc.list.gz is only 264 kB in size. Unlike etcmerge archives (from link in etcmerge manpage), it contain checksums, and checksums included for every revision, even if it was not included in official release. I think that checksums database can be even committed into CVS and will not bloat it. Ideally, this way of updating should be available even for those users who have no access to internet (distribution recieved on CDROM, etc.) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
mergemaster improvement (auto-update for not modified files)
Hello!
The mergemaster with this is test patch (attached)
can auto-update files that was not modified.
It do this by compairing each file with it's CVS
copy. If file was not modified, it can be rewritten.
This dramatically redices amount of files that require
admin's attention.
There is one major problem here:
This can be done in single-user mode only if your
have local CVS repository, because if local CVS is
not exist, anoncvs is used.
Possible solutions:
1. Pre-checkout files in pre-buildworld mode.
2. Setup CVSup collection for subset of required files
(is it possible?), and then keep this small collection
up-to-date locally.
3. Store checksum for every config file installed,
then compare with this checksum on updating.
BTW, tracking checksums can be very useful for
base system too, because checksums can help to do
clean updates (with removal of old files that was not
overwritten).
Index: mergemaster.sh
===
RCS file: /home/ncvs/src/usr.sbin/mergemaster/mergemaster.sh,v
retrieving revision 1.51
diff -u -r1.51 mergemaster.sh
--- mergemaster.sh 7 Mar 2004 10:10:19 - 1.51
+++ mergemaster.sh 3 May 2005 09:21:38 -
@@ -224,6 +224,15 @@
#
TEMPROOT='/var/tmp/temproot'
+TEMPCVS=`mktemp -q -d -t $(basename $0)`
+CVS_SKIP=''
+
+# Only one variable for the list of sites. Should be improved.
+CVS_LOGGED_IN=0
+
+CVSLIST=:pserver:[EMAIL PROTECTED]:/home/ncvs
+ :pserver:[EMAIL PROTECTED]:/home/ncvs
+
# Read /etc/mergemaster.rc first so the one in $HOME can override
#
if [ -r /etc/mergemaster.rc ]; then
@@ -871,7 +880,7 @@
# If the files have the same $Id, delete the one in temproot so the
# user will have less to wade through if files are left to merge by hand.
#
-CVSID1=`grep [$]${CVS_ID_TAG}: ${DESTDIR}${COMPFILE#.} 2/dev/null`
+CVSID1=`grep [$]${CVS_ID_TAG}: ${DESTDIR}${COMPFILE#.} 2/dev/null` ||
CVSID1=''
CVSID2=`grep [$]${CVS_ID_TAG}: ${COMPFILE} 2/dev/null` || CVSID2=none
case ${CVSID2} in
@@ -879,6 +888,60 @@
echo *** Temp ${COMPFILE} and installed have the same CVS Id, deleting
rm ${COMPFILE}
;;
+*)
+ # Checking file for changes against CVS
+ if [ ! -z ${CVSID1} -a -z ${CVS_SKIP} ]
+ then
+ CVSFILE=`echo ${CVSID1} | awk '{print $3}' | sed 's/,v$//'`
+ CVSVERSION=`echo ${CVSID1} | awk '{print $4}'`
+
+ echo *** Checking ${COMPFILE#.} against CVS
+ (cd ${TEMPCVS} cvs -Rq co -r ${CVSVERSION} ${CVSFILE} /dev/null
21) || \
+ for CVSROOT in ${CVSLIST}
+ do
+ if [ ${CVS_LOGGED_IN} -eq 0 ]; then
+ echo *** Enter 'anoncvs' password below
+ cvs -Rqd ${CVSROOT} login CVS_LOGGED_IN=1
+ fi
+ (cd ${TEMPCVS} cvs -Rqd ${CVSROOT} co -r ${CVSVERSION} ${CVSFILE}
/dev/null 21 continue)
+ done
+ case ${AUTO_RUN} in
+ '')
+ if [ $? -ne 0 ]; then
+ echo ${COMPFILE} can't be checked against CVS
+ echo -n Do you wish skip CVS checking for remaining files? y or n
[y]
+ read CVS_SKIP
+
+ echo 1 $CVS_SKIP 1
+
+ case $CVS_SKIP in
+ [Nn])
+ CVS_SKIP=''
+ ;;
+ *)
+ CVS_SKIP='y'
+ echo *** Skip CVS checking for remaining files
+ ;;
+ esac
+ fi
+ ;;
+ *)
+ echo *** Skip CVS checking for remaining files
+ ;;
+ esac
+
+
+ if diff -q ${DIFF_OPTIONS} ${DESTDIR}${COMPFILE#.}
${TEMPCVS}/${CVSFILE} \
+ /dev/null 21; then
+ if mm_install ${COMPFILE}; then
+ echo *** ${COMPFILE} installed successfully
+ else
+ echo *** Problem installing ${COMPFILE}, it will remain to merge
by hand
+ fi
+fi
+ rm -f ${TEMPCVS}/${CVSFILE}
+ fi
+;;
esac
;;
esac
@@ -926,6 +989,9 @@
echo ''
fi
+# How safe it is?
+rm -rf ${TEMPCVS}
+
case ${AUTO_RUN} in
'')
echo -n Do you wish to delete what is left of ${TEMPROOT}? [no]
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: mergemaster improvement (auto-update for not modified files)
John Hay wrote: On Tue, May 03, 2005 at 01:27:09PM +0400, Denis Peplin wrote: There is one major problem here: This can be done in single-user mode only if your have local CVS repository, because if local CVS is not exist, anoncvs is used. Possible solutions: What about mergemaster storing a copy of the original somewhere on disk for usage next time? Then the first run may still be slow, but following runs will be able to use it. Every variant based on storing some files somewhere. I'm just don't know wich variant is best (but it seems that improving mergemaster -p is easiest way). ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
