Re: [gentoo-portage-dev] Order of operations: buildpkg
Mike Frysinger wrote: On Monday 23 January 2006 15:55, Simon Stelling wrote: Lares Moreau wrote: Many ebuilds fail due to failed QA. How difficult would it be to have the package create the tarball before the QA tests. If this were possible, QA could be slightly quicker, as there would be no need to rebuild the entire package, with features disabled, upon failure. I think you rather want to use FEATURES=keepwork than doing such ugly stuff. keepwork wouldnt accomplish anything wrt to what he wants this would prob do it though: ebuild ebuild install package qmerge Indeed, could someone shade a light on what happen to /var/db/pkg and world file when using ebuild this manner ? Could be rephrased as Does it act exactly the same way emerge does ? -- gentoo-portage-dev@gentoo.org mailing list
[gentoo-portage-dev] [gentoo-dev-portage] [PATCH] prevent world file corruption by writing atomically
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi all, There is dangerous handling of world file updates throughout portage. The attached patch wraps all world file updates in a write_atomic() function which is designed to prevent interprocess interference and prevent corruption when an 'out of space' error occurs. Brian pointed out that portage_util.writedict() already does a similar routine, but it is designed exclusively to write a dictionary object. Zac -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFD1en2/ejvha5XGaMRAiWdAJ9z8qD4NAO5wj7kYnZOa8qSS1YTTACaAlFz uBy7olGkiAlYweTNv1iu0OY= =CJPG -END PGP SIGNATURE- Index: portage-2.1_pre/pym/portage_util.py === --- portage-2.1_pre.orig/pym/portage_util.py +++ portage-2.1_pre/pym/portage_util.py @@ -452,3 +452,31 @@ def unique_array(s): u.append(x) return u +def write_atomic(file_path, content): + Write a file atomically via os.rename(). Atomic replacement prevents + interprocess interference and prevents corruption of the target + file when an 'out of space' error occurs. + mystat = os.stat(file_path) + dirname,basename = os.path.split(file_path) + tmp_path=None + fd=None + try: + import tempfile + fd, tmp_path = tempfile.mkstemp(prefix=basename, dir=dirname) + os.write(fd,content) + os.close(fd) + os.chown(tmp_path, mystat.st_uid, mystat.st_gid) + os.chmod(tmp_path, mystat.st_mode) + os.rename(tmp_path,file_path) + finally: + # Make sure we've cleaned up even if an exception is raised. + if fd: + try: +os.close(fd) + except OSError, oe: +pass + if tmp_path: + try: +os.unlink(tmp_path) + except OSError, oe: +pass Index: portage-2.1_pre/pym/portage.py === --- portage-2.1_pre.orig/pym/portage.py +++ portage-2.1_pre/pym/portage.py @@ -5850,10 +5850,7 @@ class dblink: os.chown(pdir, 0, portage_gid) os.chmod(pdir, 02770) - myworld=open(self.myroot+WORLD_FILE,w) - for x in newworldlist: -myworld.write(x+\n) - myworld.close() + portage_util.write_atomic(os.path.join(self.myroot,WORLD_FILE),\n.join(newworldlist)) #do original postrm if myebuildpath and os.path.exists(myebuildpath): @@ -6862,10 +6859,7 @@ def do_upgrade(mykey): if processed: #update our internal mtime since we processed all our directives. mtimedb[updates][mykey]=os.stat(mykey)[stat.ST_MTIME] - myworld=open(/+WORLD_FILE,w) - for x in worldlist: - myworld.write(x+\n) - myworld.close() + portage_util.write_atomic(WORLD_FILE,\n.join(worldlist)) print def commit_mtimedb(): Index: portage-2.1_pre/bin/emaint === --- portage-2.1_pre.orig/bin/emaint +++ portage-2.1_pre/bin/emaint @@ -6,7 +6,7 @@ from optparse import OptionParser, Optio import re -import os, portage, portage_const +import os, portage, portage_const, portage_util class WorldHandler(object): def name(): @@ -39,7 +39,7 @@ class WorldHandler(object): def fix(self): errors = [] try: - open(portage_const.WORLD_FILE, w).write(\n.join(self.okay)) + portage_util.write_atomic(portage_const.WORLD_FILE,\n.join(self.okay)) except OSError: errors.append(portage_const.WORLD_FILE + could not be opened for writing) return errors Index: portage-2.1_pre/bin/regenworld === --- portage-2.1_pre.orig/bin/regenworld +++ portage-2.1_pre/bin/regenworld @@ -6,7 +6,7 @@ import sys sys.path.insert(0, /usr/lib/portage/pym) -import portage, string, re +import portage, portage_util, string, re __candidatematcher__ = re.compile(^[0-9]+: \\*\\*\\* emerge ) __noncandidatematcher__ = re.compile( sync( |$)| clean( |$)| search( |$)|--oneshot| unmerge( |$)) @@ -88,6 +88,4 @@ for mykey in biglist: print add to world:,myfavkey worldlist.append(myfavkey) -myfile=open(portage.WORLD_FILE, w) -myfile.write(string.join(worldlist, '\n')+'\n') -myfile.close() +portage_util.write_atomic(portage.WORLD_FILE,\n.join(worldlist)) Index: portage-2.1_pre/bin/emerge === --- portage-2.1_pre.orig/bin/emerge +++ portage-2.1_pre/bin/emerge @@ -1904,7 +1904,7 @@ class depgraph: myfavdict[myfavkey]=myfavkey print Recording,myfavkey,in \world\ favorites file... if not --fetchonly in myopts: -portage.writedict(myfavdict,portage.root+portage.WORLD_FILE,writekey=0) +portage_util.write_atomic(os.path.join(portage.root,portage.WORLD_FILE),\n.join(myfavdict.values())) portage.mtimedb[resume][mergelist]=mymergelist[:] @@ -2075,7 +2075,7 @@ class depgraph: myfavdict[myfavkey]=myfavkey print Recording,myfavkey,in \world\ favorites file... emergelog( === (+str(mergecount)+ of +str(len(mymergelist))+) Updating world file (+x[pkgindex]+)) -
[gentoo-portage-dev] confcache integration
Yo.
Looking to integrate confcache support into trunk some time in the
near future- had users testing it for about 2 months (give or take),
so far it's behaved pretty decently. A few packages eat themselves
when ran with --cache (bad autotooling), hence the addition of
restrict=confcache being added to the patch.
Either way, patch is attached, looking for any complaints regarding it
(or integrating confcache)- will be sending an email off to gentoo-dev
in the next few days to get feedback from the general dev populace,,
but sending this email to get feedback on the implementation in the
meantime.
~harring
Index: pym/portage.py
===
--- pym/portage.py (revision 2490)
+++ pym/portage.py (working copy)
@@ -2660,7 +2660,30 @@
print !!! Perhaps: rm -Rf,mysettings[BUILD_PREFIX]
print !!!,str(e)
return 1
-
+ try:
+ if confcache in features:
+ if not mysettings.has_key(CONFCACHE_DIR):
+ mysettings[CONFCACHE_DIR] =
os.path.join(mysettings[PORTAGE_TMPDIR], confcache)
+ if not
os.path.exists(mysettings[CONFCACHE_DIR]):
+
os.makedirs(mysettings[CONFCACHE_DIR], mode=0775)
+ os.chown(mysettings[CONFCACHE_DIR],
portage_uid, -1)
+ else:
+ st =
os.stat(mysettings[CONFCACHE_DIR])
+ if not (st.st_mode 0) == 0775:
+
os.chmod(mysettings[CONFCACHE_DIR], 0775)
+ if not st.st_uid == portage_uid:
+
os.chown(mysettings[CONFCACHE_DIR], portage_uid, -1)
+ for x in listdir(mysettings[CONFCACHE_DIR]):
+ p =
os.path.join(mysettings[CONFCACHE_DIR], x)
+ st = os.stat(p)
+ if not (st.st_mode 0) 07600 ==
0600:
+ os.chmod(p, (st.st_mode 0777)
| 0600)
+ if not st.st_uid == portage_uid:
+ os.chown(p, portage_uid, -1)
+
+ except OSError, e:
+ print !!! Failed resetting perms on confcachedir %s %
mysettings[CONFCACHE_DIR]
+ return 1
#try:
# mystat=os.stat(mysettings[CCACHE_DIR])
# if (mystat[stat.ST_GID]!=portage_gid) or
((mystat[stat.ST_MODE]02070)!=02070):
Index: bin/ebuild.sh
===
--- bin/ebuild.sh (revision 2490)
+++ bin/ebuild.sh (working copy)
@@ -459,7 +459,22 @@
LOCAL_EXTRA_ECONF=--libdir=${CONF_LIBDIR_RESULT}
${LOCAL_EXTRA_ECONF}
fi
- echo ${ECONF_SOURCE}/configure \
+ local TMP_CONFCACHE_DIR CONFCACHE_ARG
+ if hasq confcache $FEATURES ! hasq confcache $RESTRICT; then
+ CONFCACHE=$(type -p confcache)
+ if [ -z ${CONFCACHE} ]; then
+ ewarn disabling confcache, binary cannot be
found
+ else
+ CONFCACHE=${CONFCACHE/ /\ }
+
TMP_CONFCACHE_DIR=${CONFCACHE:+${CONFCACHE_DIR:-${PORTAGE_TMPDIR}/confcache}}
+ TMP_CONFCACHE_DIR=${TMP_CONFCACHE_DIR/ /\ }
+ CONFCACHE_ARG=--confcache-dir
+ fi
+ else
+ CONFCACHE=
+ fi
+
+ echo ${CONFCACHE} ${CONFCACHE_ARG} ${TMP_CONFCACHE_DIR}
${ECONF_SOURCE}/configure \
--prefix=/usr \
--host=${CHOST} \
--mandir=/usr/share/man \
@@ -470,7 +485,7 @@
$@ \
${LOCAL_EXTRA_ECONF}
- if ! ${ECONF_SOURCE}/configure \
+ if ! ${CONFCACHE} ${CONFCACHE_ARG} ${TMP_CONFCACHE_DIR}
${ECONF_SOURCE}/configure \
--prefix=/usr \
--host=${CHOST} \
--mandir=/usr/share/man \
pgpu5EBoUtgFS.pgp
Description: PGP signature
Re: [gentoo-portage-dev] Order of operations: buildpkg
On Tuesday 24 January 2006 03:43, Francesco Riosa wrote: Indeed, could someone shade a light on what happen to /var/db/pkg and world file when using ebuild this manner ? Could be rephrased as Does it act exactly the same way emerge does ? the 'qmerge' step would take care of updating /var/db/pkg -mike -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] Order of operations: buildpkg
On Tuesday 24 January 2006 14:19, Mike Frysinger wrote: On Tuesday 24 January 2006 03:43, Francesco Riosa wrote: Indeed, could someone shade a light on what happen to /var/db/pkg and world file when using ebuild this manner ? Could be rephrased as Does it act exactly the same way emerge does ? the 'qmerge' step would take care of updating /var/db/pkg -mike And yes it should, except that this does not clean out the /var/tmp/portage/pkgname-version directory. Use the clean command for that. If you find out the results are not the same it's a bug. (Of course package creates a package, which emerge might or might not do based on your config). The package step can also be skipped. Paul -- Paul de Vrieze Gentoo Developer Mail: [EMAIL PROTECTED] Homepage: http://www.devrieze.net pgpEaqnF93Dos.pgp Description: PGP signature
Re: [gentoo-portage-dev] Order of operations: buildpkg
On Tue, 24 Jan 2006 08:19:22 -0500 Mike Frysinger [EMAIL PROTECTED] wrote: On Tuesday 24 January 2006 03:43, Francesco Riosa wrote: Indeed, could someone shade a light on what happen to /var/db/pkg and world file when using ebuild this manner ? Could be rephrased as Does it act exactly the same way emerge does ? the 'qmerge' step would take care of updating /var/db/pkg IIRC the thing it won't do is auto-clenaing the old entries though, need to run a emerge --clean afterwards. Marius -- Public Key at http://www.genone.de/info/gpg-key.pub In the beginning, there was nothing. And God said, 'Let there be Light.' And there was still nothing, but you could see a bit better. signature.asc Description: PGP signature
Re: [gentoo-portage-dev] confcache integration
On Tue, 24 Jan 2006 04:50:31 -0800 Brian Harring [EMAIL PROTECTED] wrote: Yo. Looking to integrate confcache support into trunk some time in the near future- had users testing it for about 2 months (give or take), so far it's behaved pretty decently. A few packages eat themselves when ran with --cache (bad autotooling), hence the addition of restrict=confcache being added to the patch. Either way, patch is attached, looking for any complaints regarding it (or integrating confcache)- will be sending an email off to gentoo-dev in the next few days to get feedback from the general dev populace,, but sending this email to get feedback on the implementation in the meantime. That directory stuff really needs a wrapper function, it's duplicated way too often. Marius -- Public Key at http://www.genone.de/info/gpg-key.pub In the beginning, there was nothing. And God said, 'Let there be Light.' And there was still nothing, but you could see a bit better. signature.asc Description: PGP signature
Re: [gentoo-portage-dev] confcache integration
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian Harring wrote: Yo. Looking to integrate confcache support into trunk some time in the near future- had users testing it for about 2 months (give or take), so far it's behaved pretty decently. A few packages eat themselves when ran with --cache (bad autotooling), hence the addition of restrict=confcache being added to the patch. Either way, patch is attached, looking for any complaints regarding it (or integrating confcache)- will be sending an email off to gentoo-dev in the next few days to get feedback from the general dev populace,, but sending this email to get feedback on the implementation in the meantime. ~harring I don't see a manpage/make.conf patch in there :) -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQ9YzA2zglR5RwbyYAQKCTg/+NtqHZSIrApdkm6+TULFE2W2CIHQD9Vd8 VdLyKaDXOigsK460iZehY1pnxZRA8qes9Ed4H51kWL5HMCO/9FHyUZq2SAYHwQ/M moGMML+XyPanUc9Kvy2JREc/onzCbvPdYXkcJRNuE2XFhrfnKTXWin8JJBqyP7C/ 9S/NItRH+lAE6/1Az/6iJcSto0WscN/W18shwitCUB0igVUYcaBal4W3oJ6SvHEB NtX/xjL6zHTlS1iCsawVXAzgwcvFBNOSWilFobN6Qj2V1cn9a74Pio4axl33pEzK 6Mz4csL/UCZb/RisABOU03i3tTBEnehwgvKaD1BxHVfTUNwr8f9Oc1GdLJoZKlp8 H+4mZ7K82bhJ8CB0io9Zz5FZFUrp2cU2mQCh53vlVHZ2UlH+rAXaCLp2Lm0Fi1yQ 5cYyp0tH4eGNrYBrykZZ/gImaGYhIJ2RR8AETROltzRLk9bXq5dMgvWF2+38DdBI RqJKMYMmHSYJ/Ccp0kLs4G6xhOw5VXpRIdPMOFPOh76szdJX801s8S0ZHWAPcc7q /lkfnc9JwKcLSQIGth0IllEfhe3/OtaZoi+SDV0hIuOwJ0AgesUzNnUIa41pAfW5 npGTImc8kXYSp1JoDyflpUgsY6Cv2w48K96QUTWpj5oOPtu4l2t0wsof0boekFjP BR12OBv/wB4= =cCKG -END PGP SIGNATURE- -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] Order of operations: buildpkg
Paul de Vrieze wrote: On Tuesday 24 January 2006 14:19, Mike Frysinger wrote: On Tuesday 24 January 2006 03:43, Francesco Riosa wrote: Indeed, could someone shade a light on what happen to /var/db/pkg and world file when using ebuild this manner ? Could be rephrased as Does it act exactly the same way emerge does ? the 'qmerge' step would take care of updating /var/db/pkg -mike And yes it should, except that this does not clean out the /var/tmp/portage/pkgname-version directory. Use the clean command for that. If you find out the results are not the same it's a bug. (Of course package creates a package, which emerge might or might not do based on your config). The package step can also be skipped. Paul That's fine, these functions are intended for developer use that may need to still have the work/image dir in place. thanks everyone to address my worries . Francesco -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] making aux_get more usable for vardbapi
On Mon, 23 Jan 2006 14:08:00 -0800
Brian Harring [EMAIL PROTECTED] wrote:
On Mon, Jan 23, 2006 at 07:44:44PM +0100, Marius Mauch wrote:
On Mon, 23 Jan 2006 03:47:11 -0800
Can't follow your thinking here. As said, the code won't corrupt any
data, at worst it will tell the user that it couldn't extract some
keys (and even that only if there would be such a stupid case which
itself has a chance of like 10^-10 or so, or can you name a single
case?).
Yeah, any extension of your code to pick up EAPI is going to false
positive on somewhere around a quarter of my vdb nodes- I use a
debug function from my bashrc for prefix testing (holdover from eapi
testing days). Why? Because the eapi var didn't exist (thus no env
assignment), the only possible match is in the middle of a here op-
debug_dump() {
...
echo HERE 2
EAPI=${EAPI-unset}
PF=$PF
CATEGORY=$CATEGORY
HERE
...
}
Setting EAPI to unset strikes me as corruption here, because now
portage will *never* do anything with that vdb node- the eapi differs
from what it supports (it's been set to effectively a random value),
thus I have to fix the vdb entry myself if I ever want to get rid of
it.
Guessing the response on that one is well, you're using the bashrc,
and yes, I am- the vdb code should be *robust*, not choking on a
users debug code.
Not even trying to comment on this one.
So what package contains filter-env then?
Portage- been in cvs/svn for over a year now :)
Not in trunk or any release (except the alpha snapshot). And I'm not
going to blindly copy unknown code from completely different codebases
and later be blamed for doing it improperly.
Aside from that, if the code is in debate (as this is), I really
don't think it should get slid into svn 2 weeks later effectively
unchanged- didn't write that original email just for the hell of
it :)
As said, I disagree with your assessment of the situation. If you
can name a single case where the code breaks or filter-env hits
the tree I might reconsider, but not before.
Well, if you disagreed with the original response, continue the
conversation prior to commiting- otherwise we see a commit, then a
rebuttal a few hours later. Not really how things should go for a
contested piece of code (at least when the only two to weigh in our
flat out opposed on it)- especially if the code's effect is
nontrivial and it hasn't had any actual peer review (only comment was
on your algo).
Interesting, you now count for two people? Or who is this second person
you're talking of? I still think you're making more out of this than it
really is. Also there really isn't a point in discussing a difference in
opinion IMO, such attempts lead nowhere. I considered your comment, but
couldn't come up with a reason why a theoretical issue should hold this
up.
Issues with the code.
Now we're getting somewhere.
1) Scans for metadata that didn't exist at the time of the ebuild's
installation, only possible match is unintended. Not good.
Not really sure what that's supposed to mean, assuming you mean that
nodes will lack SRC_URI/HOMEPAGE/... from env.bz2 is about as likely as
here doc statements containing them (and indicates a broken package
anyway).
2) your code is breaking upon *all* newlines, regardless if it's
in the middle of a variable assignment. Yes, bash does use $'' for
assignment, but you're relying (hoping really) that the variable has
been filtered by python portage and the ebuild defined var is
stomped. This is a bad idea- description comes to mind. Your code
seems to be aware of this possibility also, since it's doing rather
loose quote filtering.
Have to think about this.
3) Stop using regex all over the place. split/join is a helluva lot
faster, and collapses that nasty regex that attempts to remove
whitespace down to two lines (a one time translate, and the
split/join to kill whitespace).
Ok (cosmetics).
4) Code specifically tries to find, and remove variable chars-
this is *really* the wrong thing to do. If you're trying to work
around catching a var reference, either your parsing screwed up, or
you're mangling a DESCRIPTION value that you shouldn't be hosing.
define variable chars, otherwise doesn't make sense.
5) hardcoded vdb.
6) Non root aware VDB.
Ok (trivial)
7) Only is aware of environment.bz2- older portage versions differed
in this afaik (I *do* know PORT_ENV_FILE overloading for ebd I had to
address this case already, so it's out there).
Well, it's present as long as I can remember, but an sanity check
wouldn't hurt there.
8) perms on the new files?
Ok.
9) general code comments; 'if s != ', use 'if s:'. Catch the
exceptions for check mode, don't let vdbkeyhandler nuke world file
checking due to a potential exception. Right now, your code will TB
if ran by non-root for a check run rather then stating got to be
root.
a) hate theses
Re: [gentoo-portage-dev] confcache integration
On Tuesday 24 January 2006 21:50, Brian Harring wrote: +os.makedirs(mysettings[CONFCACHE_DIR], mode=0775) +os.chown(mysettings[CONFCACHE_DIR], portage_uid, -1) This will die when running as non-root, no? ebuild is now oft used by devs running as normal users which will be broken by this. No clues on the bash stuff; it seems there's an external confcache binary but I can't tell much beyond that. -- Jason Stubbs -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [gentoo-dev-portage] [PATCH] prevent world file corruption by writing atomically
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian Harring wrote: the atomic writing of data in writedict could be gutted out to a derivative of the file class; via that, same underlying atomic update code for writedict and wordfile updates... Hmm, override the constructor and close methods to handle the temp file creation and rename, respectively. That seems like a very elegant approach. I'll try that and update the patch. Zac -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFD1nHS/ejvha5XGaMRAhcCAJ4v2ZoUux1lqDpq8z9URZfybZg2IgCfVwDK XPPGuL0jiv3RgO1xPCsGFt0= =E4NK -END PGP SIGNATURE- -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] confcache integration
On Wed, 2006-01-25 at 00:30 +0900, Jason Stubbs wrote: On Tuesday 24 January 2006 21:50, Brian Harring wrote: +os.makedirs(mysettings[CONFCACHE_DIR], mode=0775) +os.chown(mysettings[CONFCACHE_DIR], portage_uid, -1) This will die when running as non-root, no? ebuild is now oft used by devs running as normal users which will be broken by this. Your right. I've hit that bug myself in the past switching between root merges and being a real life dev and running ebuild foo.build clean unpack compile; -- solar [EMAIL PROTECTED] Gentoo Linux -- gentoo-portage-dev@gentoo.org mailing list
