Re: [gentoo-dev] Re: metadata/md5-cache
On Sun, 3 Jun 2012 09:48:26 + "Robin H. Johnson" wrote: > On Sun, Jun 03, 2012 at 11:34:07AM +0200, Micha?? G??rny wrote: > > I means using separate proto for metadata, not necesarrily git. In > > any case, if it comes to transferring a lot of frequently-changing > > files, rsync is not that efficient... > It does NOT send any of the intermediate states. But it does have to check all the files. Did I mention I'm not talking necessarily about git? Rather anything which would just lookup our timestamp, revision or whatever and just send what have changed, in a packed manner. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: Add new remote-id types in metadata.dtd
On Sat, Jun 2, 2012 at 7:30 PM, Michael Weber wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Is there any way to verify the data? > > What programs/scripts use these fields btw? euscan will soon, and I guess p.g.o will too. > I could imagine a test like (i.e. type="sourceforge">$a ) > does http://www.fs.net/$a exist. > > Michael That could probably be done in remoteids.py
[gentoo-dev] [PATCH vcs-snapshot] Use ${WORKDIR}/${P} rather than ${S} to support ${S} overrides.
One could set S to work on a subtree of the tarball rather than
the whole tarball. Considering that, it's probably better to use
${WORKDIR}/${P} rather than ${S}.
Fixes: https://bugs.gentoo.org/show_bug.cgi?id=419479
---
gx86/eclass/vcs-snapshot.eclass |5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/gx86/eclass/vcs-snapshot.eclass b/gx86/eclass/vcs-snapshot.eclass
index 6748360..23cd696 100644
--- a/gx86/eclass/vcs-snapshot.eclass
+++ b/gx86/eclass/vcs-snapshot.eclass
@@ -9,7 +9,8 @@
# @DESCRIPTION:
# This eclass provides a convenience src_unpack() which does support
# working with snapshots generated by various VCS-es. It unpacks those
-# to ${S} rather than the original directory containing commit id.
+# to ${WORKDIR}/${P} rather than the original directory containing
+# commit id.
#
# Note that this eclass handles only unpacking. You need to specify
# SRC_URI yourself, and call any autoreconfiguration as necessary.
@@ -41,5 +42,5 @@ vcs-snapshot_src_unpack() {
# github, bitbucket: username-projectname-hash
# gitweb: projectname-tagname-hash
- mv *-*-[0-9a-f]*[0-9a-f]/ "${S}" || die
+ mv *-*-[0-9a-f]*[0-9a-f]/ "${WORKDIR}"/${P} || die
}
--
1.7.10.2
[gentoo-dev] [java-utils-2.eclass] - removal of java-pkg_ensure-test and java-pkg_ensure-gcj
Both java-pkg_ensure-test and java-pkg_ensure-gcj will be removed from
java-utils-2.eclass after the 4 of July 2012. See attached patch.
java-pkg_ensure-test: [1]
Was used to enforce USE=test if FEATURES=test. For quite some time this
is handled by package managers. Relies on the env var FEATURES [2].
There are no known consumers any more.
Solution: Remove the call to java-pkg_ensure-test and rely on the
package manager handling this.
java-pkg_ensure-gcj:
Uses built_with_use [3] and ebeep [4] and is no longer needed with EAPI
2 or later. There are no known consumers for quite some time.
Solution: Migrate to EAPI 2 or later and depend on sys-devel/gcc[gcj]
instead.
[1] https://bugs.gentoo.org/show_bug.cgi?id=278965
[2] https://bugs.gentoo.org/show_bug.cgi?id=174335
[3] https://bugs.gentoo.org/show_bug.cgi?id=261562
[4] https://bugs.gentoo.org/show_bug.cgi?id=377943
Index: java-utils-2.eclass
===
RCS file: /var/cvsroot/gentoo-x86/eclass/java-utils-2.eclass,v
retrieving revision 1.150
diff -u -b -B -r1.150 java-utils-2.eclass
--- java-utils-2.eclass 13 Mar 2012 10:05:46 - 1.150
+++ java-utils-2.eclass 4 Jun 2012 10:15:57 -
@@ -1686,23 +1686,13 @@
}
java-pkg_ensure-gcj() {
- if ! built_with_use sys-devel/gcc gcj ; then
- ewarn
- ewarn "You must build gcc with the gcj support to build with gcj"
- ewarn
- ebeep 5
- die "No GCJ support found!"
- fi
+ # was enforcing sys-devel/gcc[gcj}
+ die "${FUNCNAME} was removed. Use use-deps available as of EAPI 2 instead. #261562"
}
java-pkg_ensure-test() {
- if has test ${FEATURES} && ! has -test ${FEATURES} \
- && has test ${IUSE} && ! use test;
- then
- eerror "You specified FEATURES=test, but USE=test is needed"
- eerror "to pull in the additional dependencies for testing"
- die "Need USE=test enabled"
- fi
+ # was enforcing USE=test if FEATURES=test
+ die "${FUNCNAME} was removed. Package mangers handle this already. #278965"
}
# --
signature.asc
Description: PGP signature
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 2:50 AM, Dirkjan Ochtman wrote: > On Sun, Jun 3, 2012 at 9:35 PM, Andreas K. Huettel > wrote: >> However, then the "committer" of the contributed commits before the merge is >> then the user, I guess? >> >> (The rule meaning as suggested by Robin >>> - if you include a commit from a user: >>> author := non-@gentoo >>> committer := @gentoo >>> signer := $committer > > I guess, I'm not sure how the committer thing works in git. > Well, only Robin can explain exactly what he meant, but it sounds like we don't want the committer field to ever have a non-gentoo email in it, and signatures should be gentoo as well. So, if a dev just applies a patch to their tree/etc then there is no issue (just set author). If a dev wants to actually pull in a commit they'd need to edit the fields accordingly and re-sign it. Not sure offhand how to best do that (I assume it is possible - probably with some variation on rebase or something rebase calls). I don't think the intent is to snub non-devs. The issue is what is the purpose of the signatures and committers field in the first place. The signature verifies that the commit is intact, and you can only do that if you have a key to check it with, and you can trust that key. If the signer is a dev then we already have policy that the keys need to be published, and we have a list of key IDs on our website. I'm sure that could be improved on. If we stick non-dev signatures in the tree then that becomes more of a problem (though it clearly is possible - maybe something to think about). I assume the committer denotes a layer of accountability, and having a dev in that spot makes sense (devs who are proxies are accountable for oversight at some level - though I'd personally give them the benefit of the doubt since we want to encourage the proxy role). I think the key with git is to not let the perfect be the enemy of the good. We don't have an unbroken signature chain on our current portage tree, so I don't think we need one to move to git. As long as git is at least as good as what we have now, then we should accept it. We should of course strive to improve, but let's not keep the almost completely unsigned cvs around for another 10 years while we argue about signatures. Rich
Re: [gentoo-dev] Git braindump: 2 of N: developer interaction (merge co-ordinators)
On Mon, Jun 4, 2012 at 2:48 AM, Dirkjan Ochtman wrote: > IMO we should try to be cutting down barriers from the git migration, > not throwing up more. The process has taken long enough already; the > desire to control everything about the migration is part of why it's > taken so long. Fair enough. We should also be asking ourselves what happens if something goes really wrong and there are no commits for a whole two days. I think the answer is, not much. As long as there are a few devs who can tend to security issues or whatever I don't see it as an issue if we have the odd glitch. We do need to get the docs up so that we're at least somewhat harmonized on conventions (authors vs committers, and so on). Let's not burn the first person to make a mistake at the stake either... Rich
Re: [gentoo-dev] Git braindump: 2 of N: developer interaction (merge co-ordinators)
On Mon, Jun 4, 2012 at 2:38 PM, Rich Freeman wrote: > On Mon, Jun 4, 2012 at 2:48 AM, Dirkjan Ochtman wrote: >> IMO we should try to be cutting down barriers from the git migration, >> not throwing up more. The process has taken long enough already; the >> desire to control everything about the migration is part of why it's >> taken so long. > > Fair enough. > > We should also be asking ourselves what happens if something goes > really wrong and there are no commits for a whole two days. I think > the answer is, not much. As long as there are a few devs who can tend > to security issues or whatever I don't see it as an issue if we have > the odd glitch. That's what I was thinking, too. :) > We do need to get the docs up so that we're at least somewhat > harmonized on conventions (authors vs committers, and so on). Let's > not burn the first person to make a mistake at the stake either... Yeah, some initial tutorial of how to get going would probably help a lot. For everything else, there's IRC (and hopefully most of what's discussed there would soon find it's way into the HOWTO doc). Cheers, Dirkjan
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 2:34 PM, Rich Freeman wrote: > Well, only Robin can explain exactly what he meant, but it sounds like > we don't want the committer field to ever have a non-gentoo email in > it, and signatures should be gentoo as well. So, if a dev just > applies a patch to their tree/etc then there is no issue (just set > author). If a dev wants to actually pull in a commit they'd need to > edit the fields accordingly and re-sign it. Not sure offhand how to > best do that (I assume it is possible - probably with some variation > on rebase or something rebase calls). Well, it doesn't seem like a big deal IF there's an explicit merge commit that's signed by a dev. Cheers, Dirkjan
[gentoo-dev] Re: [gentoo-commits] gentoo-x86 commit in x11-themes/gtk-engines-nodoka: ChangeLog
Something went wrong here. On 06/04/2012 03:39 PM, Alex Legler (a3li) wrote: a3li12/06/04 12:39:01 Modified: ChangeLog Log: Drop maintainership (Portage version: 2.2.0_alpha103/cvs/Linux x86_64) Revision ChangesPath 1.6 x11-themes/gtk-engines-nodoka/ChangeLog file : http://sources.gentoo.org/viewvc.cgi/gentoo-x86/x11-themes/gtk-engines-nodoka/ChangeLog?rev=1.6&view=markup plain: http://sources.gentoo.org/viewvc.cgi/gentoo-x86/x11-themes/gtk-engines-nodoka/ChangeLog?rev=1.6&content-type=text/plain diff : http://sources.gentoo.org/viewvc.cgi/gentoo-x86/x11-themes/gtk-engines-nodoka/ChangeLog?r1=1.5&r2=1.6 Index: ChangeLog === RCS file: /var/cvsroot/gentoo-x86/x11-themes/gtk-engines-nodoka/ChangeLog,v retrieving revision 1.5 retrieving revision 1.6 diff -u -r1.5 -r1.6 --- ChangeLog 5 May 2012 04:10:06 - 1.5 +++ ChangeLog 4 Jun 2012 12:39:00 - 1.6 @@ -1,6 +1,9 @@ # ChangeLog for x11-themes/gtk-engines-nodoka # Copyright 1999-2012 Gentoo Foundation; Distributed under the GPL v2 -# $Header: /var/cvsroot/gentoo-x86/x11-themes/gtk-engines-nodoka/ChangeLog,v 1.5 2012/05/05 04:10:06 jdhore Exp $ +# $Header: /var/cvsroot/gentoo-x86/x11-themes/gtk-engines-nodoka/ChangeLog,v 1.6 2012/06/04 12:39:00 a3li Exp $ + + 04 Jun 2012; Alex Legler ChangeLog: + Drop maintainership 05 May 2012; Jeff Horelick gtk-engines-nodoka-0.7.2.ebuild, gtk-engines-nodoka-0.7.5.ebuild:
Re: [gentoo-dev] Re: metadata/md5-cache
On Mon, Jun 04, 2012 at 09:27:10AM +0200, Micha?? G??rny wrote: > On Sun, 3 Jun 2012 09:48:26 + > "Robin H. Johnson" wrote: > > > On Sun, Jun 03, 2012 at 11:34:07AM +0200, Micha?? G??rny wrote: > > > I means using separate proto for metadata, not necesarrily git. In > > > any case, if it comes to transferring a lot of frequently-changing > > > files, rsync is not that efficient... > > It does NOT send any of the intermediate states. > > But it does have to check all the files. Which is a pretty minimal cost in the grand scheme of things. You also need to figure out what 'efficiency' you're going to talk about here; network io, disk io, cpu io, etc. Most people in this case care about network IO; rsync's not perfect, but for reasons described below, it's the best of breed for the usage scenario. > Did I mention I'm not talking necessarily about git? Git would be sanest if you were after this; it already does point to point delta transformations sanely. No point in reinventing a VCS; if you can't force the tree back to a known good state (aka, distributed VCS), you can't apply deltas to it, which case you need an rsync like algo. > Rather anything which would just > lookup our timestamp, revision or whatever and just send what have > changed, in a packed manner. This would be reinventing git/VCS, or more likely, pretending that a timestamp file automatically means the repository is *unmodified*, and trying to do a point to point transformation on it. Where you're notion breaks down is that fun little bit about "unmodified". This is why rsync is used; it's not limited to a point to point transformation, it's able to work from any starting point *efficiently*. Either way, suggest you do some research into this- including efficiencies of rsync, git, existing snapshot delta rsync machinery (tarsync, diffball, etc), study the trade offs inherint in each. Your initial email frankly reaks of NIH, hence my suggestions to go investigate what exists now. ~harring
Re: [gentoo-dev] Git braindump: 2 of N: developer interaction (merge co-ordinators)
On Mon, Jun 04, 2012 at 03:49:31AM +, Kent Fredric wrote: > On 3 June 2012 09:46, Robin H. Johnson wrote: > If there are enough "Alice" developers, is it a possibility that Bob > > will never have a chance to get his commit in? > > > > All this requires, is that in the time it takes Bob to do 'git pull', > > Alice manages to do 'git push' again. > > > > Alice can thus deprive Bob of a fair chance to get his commit in. > > Bob becomes an unhappy developer and gives up. > > There's an easier solution here: > > Bob pushes to a branch or to a public repo ( ie: github ) , and then > contacts Alice ( or somebody else ) who pulls their changes into the > tree on their behalf. > > Its not "ideal" but better than nothing. And certainly better than > being stuck on SVN where this case is virtually guaranteed and with no > viable workarounds when it is encountered. Kent, you did read Robin's email fully before commenting, right? ;) You just proposed 'merge lieutenants', which Robin already covered in the originating email of this thread: http://archives.gentoo.org/gentoo-dev/msg_f478e9cbb14feb01ad0771c5d24222c4.xml For the record, I'm against any form of merge lieutenant reliant on someone pulling shit in; automated (QA of some form) I'd be fine w/, although that's not simple machinery to slap into the proposals. While I do grok the potential issue of someone being a hog (specifically via blasting commit by commit rather than building up work locally, then pushing it in chunks), frankly... I'm not that concerned about it, and would rather deal w/ it if/when it occurs. The nature of our commits for the most part are standalone from others- that's not true of the kernel/mozilla, thus why I don't think their issues are necessarily ours. ~harring
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 8:45 AM, Dirkjan Ochtman wrote: > > Well, it doesn't seem like a big deal IF there's an explicit merge > commit that's signed by a dev. I'm not sure about that. If you were verifying a tree, how would you identify which commits were merged in by what dev, using an automated algorithm? The only thing the merge commit contains is a list of two parents, and a tree. It doesn't say which one is which, unless we can rely on their order. Now, all those intermediate commits were never actually published via rsync, so their integrity isn't a direct issue. However, I'm not sure how easy automated verification would be. Rich
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 3:40 PM, Rich Freeman wrote: > The only thing the merge commit contains is a list of two parents, and > a tree. It doesn't say which one is which, unless we can rely on > their order. You simply walk the tree from root to tip. When you encounter an unsigned changeset, the nearest signed descendant is responsible for merging that changeset. Cheers, Dirkjan
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On 06/04/2012 07:34 AM, Rich Freeman wrote: > On Mon, Jun 4, 2012 at 2:50 AM, Dirkjan Ochtman wrote: >> On Sun, Jun 3, 2012 at 9:35 PM, Andreas K. Huettel >> wrote: >>> However, then the "committer" of the contributed commits before the merge is >>> then the user, I guess? >>> >>> (The rule meaning as suggested by Robin - if you include a commit from a user: author := non-@gentoo committer := @gentoo signer := $committer >> >> I guess, I'm not sure how the committer thing works in git. >> > > Well, only Robin can explain exactly what he meant, but it sounds like > we don't want the committer field to ever have a non-gentoo email in > it, and signatures should be gentoo as well. So, if a dev just > applies a patch to their tree/etc then there is no issue (just set > author). If a dev wants to actually pull in a commit they'd need to > edit the fields accordingly and re-sign it. Not sure offhand how to > best do that (I assume it is possible - probably with some variation > on rebase or something rebase calls). > > I don't think the intent is to snub non-devs. The issue is what is > the purpose of the signatures and committers field in the first place. > The signature verifies that the commit is intact, and you can only do > that if you have a key to check it with, and you can trust that key. > If the signer is a dev then we already have policy that the keys need > to be published, and we have a list of key IDs on our website. I'm > sure that could be improved on. If we stick non-dev signatures in the > tree then that becomes more of a problem (though it clearly is > possible - maybe something to think about). I assume the committer > denotes a layer of accountability, and having a dev in that spot makes > sense (devs who are proxies are accountable for oversight at some > level - though I'd personally give them the benefit of the doubt since > we want to encourage the proxy role). > > I think the key with git is to not let the perfect be the enemy of the > good. We don't have an unbroken signature chain on our current > portage tree, so I don't think we need one to move to git. As long as > git is at least as good as what we have now, then we should accept it. > We should of course strive to improve, but let's not keep the almost > completely unsigned cvs around for another 10 years while we argue > about signatures. > > Rich > I think the intent is to only have commits and signoffs come from @gentoo, but we need a way to give attribution to users who send stuff in that gets committed. -- -- Matthew Thode (prometheanfire) signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 9:48 AM, Dirkjan Ochtman wrote: > > You simply walk the tree from root to tip. When you encounter an > unsigned changeset, the nearest signed descendant is responsible for > merging that changeset. > How do you KNOW that the nearest signed descendant actually merged it? How do you know it wasn't added by a hacker? Also, when walking the tree keep in mind that there isn't just one path in it (with merge commits), and the links are from any particular HEAD going back. I'm not convinced that this is impossible, but it isn't as trivial as it might seem at first glance. Rich
Re: [gentoo-dev] [PATCH vcs-snapshot] Use ${WORKDIR}/${P} rather than ${S} to support ${S} overrides.
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 06/04/2012 11:59 AM, Michał Górny wrote:
> One could set S to work on a subtree of the tarball rather than the
> whole tarball. Considering that, it's probably better to use
> ${WORKDIR}/${P} rather than ${S}.
>
> Fixes: https://bugs.gentoo.org/show_bug.cgi?id=419479 ---
> gx86/eclass/vcs-snapshot.eclass |5 +++-- 1 file changed, 3
> insertions(+), 2 deletions(-)
>
> diff --git a/gx86/eclass/vcs-snapshot.eclass
> b/gx86/eclass/vcs-snapshot.eclass index 6748360..23cd696 100644 ---
> a/gx86/eclass/vcs-snapshot.eclass +++
> b/gx86/eclass/vcs-snapshot.eclass @@ -9,7 +9,8 @@ # @DESCRIPTION: #
> This eclass provides a convenience src_unpack() which does support
> # working with snapshots generated by various VCS-es. It unpacks
> those -# to ${S} rather than the original directory containing
> commit id. +# to ${WORKDIR}/${P} rather than the original directory
> containing +# commit id. # # Note that this eclass handles only
> unpacking. You need to specify # SRC_URI yourself, and call any
> autoreconfiguration as necessary. @@ -41,5 +42,5 @@
> vcs-snapshot_src_unpack() {
>
> # github, bitbucket: username-projectname-hash # gitweb:
> projectname-tagname-hash -mv *-*-[0-9a-f]*[0-9a-f]/ "${S}" || die
> + mv *-*-[0-9a-f]*[0-9a-f]/ "${WORKDIR}"/${P} || die }
Currently that eclass does not support multiple snapshot sources/tarballs.
I would suggest to fix that. My proposal currently breaks 6 ebuilds
which do not follow the newly described standard.
This way you could merge data and source snapshots from vcs.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJPzMQSAAoJEFpvPKfnPDWzJdMH/AuhgbDeVdnJViP+zZNjsq4B
C2FlaPCpl3HHTrMi7HhbFmbOnZeLA3VLaffdvaPvZU0O1RxhOQPNfp4t8/E2iuHF
q9vVwQFQe19izFa7EgapD2EOUVUeaNMGoX0K2qRgtfLeBwnLWsnagMYZkV2+5skA
84vnyRYtaDEx6Z+M0UipwFnubrgmItbvX7F3XZ3omCdlSjXaRDO0sc2hqMFjeZ5P
ESGfbWxFX59ztmxBni0iSJE3NdSiYr1mjuW6eFZazIq9T6cinByXfMJgbdz7N14e
8HaSxMTc6PVWUTVbmpj3POWnZ+KCHbFze39rO5HzY4ZPMIL4xTxXAppg2FnyhF4=
=KUJi
-END PGP SIGNATURE-
--- vcs-snapshot.eclass.old 2012-06-04 16:10:25.0 +0200
+++ vcs-snapshot.eclass 2012-06-04 16:11:16.0 +0200
@@ -8,8 +8,10 @@
# @BLURB: support eclass for VCS (github, bitbucket, gitweb) snapshots
# @DESCRIPTION:
# This eclass provides a convenience src_unpack() which does support
-# working with snapshots generated by various VCS-es. It unpacks those
-# to ${S} rather than the original directory containing commit id.
+# working with snapshots generated by various VCS-es. It unpacks all tarballs
+# and zipballs according to their name into ${WORKDIR}/${name}
+# rather than the original directory containing commit id.
+# See example below.
#
# Note that this eclass handles only unpacking. You need to specify
# SRC_URI yourself, and call any autoreconfiguration as necessary.
@@ -26,6 +28,10 @@
# inherit autotools-utils vcs-snapshot
#
# SRC_URI="http://github.com/example/${PN}/tarball/v${PV} -> ${P}.tar.gz"
+#
+# # line above unpacks into: ${WORKDIR}/${P}
+# # SRC_URI="http://github.com/example/${PN}/tarball/v${PV} -> ${P}-src.tar.gz"
+# # would unpack into: ${WORKDIR}/${P}-src
# @CODE
case ${EAPI:-0} in
@@ -37,9 +43,21 @@
EXPORT_FUNCTIONS src_unpack
vcs-snapshot_src_unpack() {
- default
-
# github, bitbucket: username-projectname-hash
# gitweb: projectname-tagname-hash
- mv *-*-[0-9a-f]*[0-9a-f]/ "${S}" || die
+ local i
+ local _tmpdir=${T}/_tmp
+ mkdir "${_tmpdir}" || die
+ pushd "${_tmpdir}" > /dev/null || die
+ for i in ${A} ; do
+ unpack ${i}
+ case ${i} in
+ *.tar.gz) mv "${_tmpdir}"/*/ "${WORKDIR}"/${i%.tar.gz} || die ;;
+ *.tar.xz) mv "${_tmpdir}"/*/ "${WORKDIR}"/${i%.tar.xz} || die ;;
+ *.tar.bz2) mv "${_tmpdir}"/*/ "${WORKDIR}"/${i%.tar.bz2} || die ;;
+ *.zip) mv "${_tmpdir}"/*/ "${WORKDIR}"/${i%.zip} || die ;;
+ *) mv "${_tmpdir}"/* "${WORKDIR}"/ ;;
+ esac
+ done
+ popd > /dev/null
}
# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/eclass/vcs-snapshot.eclass,v 1.2 2012/03/19
08:52:49 mgorny Exp $
# @ECLASS: vcs-snapshot.eclass
# @MAINTAINER:
# mgo...@gentoo.org
# @BLURB: support eclass for VCS (github, bitbucket, gitweb) snapshots
# @DESCRIPTION:
# This eclass provides a convenience src_unpack() which does support
# working with snapshots generated by various VCS-es. It unpacks all tarballs
# and zipballs according to their name into ${WORKDIR}/${name}
# rather than the original directory containing commit id.
# See example below.
#
# Note that this eclass handles only unpacking. You need to specify
# SRC_URI yourself, and call any autoreconfiguration as necessary.
# The example does that using autotools-utils eclass.
#
# Right now, the eclass was tested with github, bitbucket and gitweb
# snapshots. Feel free to report snapshotting services which ar
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 4:18 PM, Rich Freeman wrote: > How do you KNOW that the nearest signed descendant actually merged it? > > How do you know it wasn't added by a hacker? Because then the signature for the nearest signed descendant wouldn't check out (unless it got hacked before he signed it, of course, but in that case hopefully he wouldn't sign it...). > Also, when walking the tree keep in mind that there isn't just one > path in it (with merge commits), and the links are from any particular > HEAD going back. I'm not convinced that this is impossible, but it > isn't as trivial as it might seem at first glance. Well, this only means there might potentially be multiple nearest signed descendants, but I don't think that's a problem. Feel free to shoot holes in it, but I think this checks out. Of course, we'd have to make sure the tip of whatever is pushed is always signed, but the hook for that should be trivial. Cheers, Dirkjan
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 10:26 AM, Dirkjan Ochtman wrote: > On Mon, Jun 4, 2012 at 4:18 PM, Rich Freeman wrote: >> How do you KNOW that the nearest signed descendant actually merged it? >> >> How do you know it wasn't added by a hacker? > > Because then the signature for the nearest signed descendant wouldn't > check out (unless it got hacked before he signed it, of course, but in > that case hopefully he wouldn't sign it...). When I do a cvs commit, I don't check the logs to make sure the last 25 commits all look valid. So, why would I expect others to do any differently in git. I make my changes, I run a git pull (bringing in the hacked commit on gentoo-x86 master), and then merge/rebase in my changes, signing my commit (which indicates that what _I_ just commited is good, not that everything before is good). I am not the one commiting in hacked files - they were there before I got there. > > Of course, we'd have to make sure the tip of whatever is pushed is > always signed, but the hook for that should be trivial. Yup, but the hacker wouldn't run the hook. Rich
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 4:48 PM, Rich Freeman wrote: > When I do a cvs commit, I don't check the logs to make sure the last > 25 commits all look valid. So, why would I expect others to do any > differently in git. I make my changes, I run a git pull (bringing in > the hacked commit on gentoo-x86 master), and then merge/rebase in my > changes, signing my commit (which indicates that what _I_ just > commited is good, not that everything before is good). I am not the > one commiting in hacked files - they were there before I got there. If the tree was bad before you pushed, then it's not your fault the tree is bad. You're only responsible for the commits you bring into the tree, so if you're merging contributor's unsigned changesets, you merge them with a signature of your own. >> Of course, we'd have to make sure the tip of whatever is pushed is >> always signed, but the hook for that should be trivial. > > Yup, but the hacker wouldn't run the hook. If the hacker has unfettered access to the server where the repository lives, we probably have bigger problems, as they can get whatever rsynced to all our users. I guess we could have rsync process check that the cset it's about to push out to mirrors is signed? Cheers, Dirkjan
Re: [gentoo-dev] [PATCH vcs-snapshot] Use ${WORKDIR}/${P} rather than ${S} to support ${S} overrides.
On Mon, 04 Jun 2012 16:20:02 +0200
hasufell wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 06/04/2012 11:59 AM, Michał Górny wrote:
> > One could set S to work on a subtree of the tarball rather than the
> > whole tarball. Considering that, it's probably better to use
> > ${WORKDIR}/${P} rather than ${S}.
> >
> > Fixes: https://bugs.gentoo.org/show_bug.cgi?id=419479 ---
> > gx86/eclass/vcs-snapshot.eclass |5 +++-- 1 file changed, 3
> > insertions(+), 2 deletions(-)
> >
> > diff --git a/gx86/eclass/vcs-snapshot.eclass
> > b/gx86/eclass/vcs-snapshot.eclass index 6748360..23cd696 100644 ---
> > a/gx86/eclass/vcs-snapshot.eclass +++
> > b/gx86/eclass/vcs-snapshot.eclass @@ -9,7 +9,8 @@ # @DESCRIPTION: #
> > This eclass provides a convenience src_unpack() which does support
> > # working with snapshots generated by various VCS-es. It unpacks
> > those -# to ${S} rather than the original directory containing
> > commit id. +# to ${WORKDIR}/${P} rather than the original directory
> > containing +# commit id. # # Note that this eclass handles only
> > unpacking. You need to specify # SRC_URI yourself, and call any
> > autoreconfiguration as necessary. @@ -41,5 +42,5 @@
> > vcs-snapshot_src_unpack() {
> >
> > # github, bitbucket: username-projectname-hash # gitweb:
> > projectname-tagname-hash - mv *-*-[0-9a-f]*[0-9a-f]/ "${S}"
> > || die
> > + mv *-*-[0-9a-f]*[0-9a-f]/ "${WORKDIR}"/${P} || die }
>
> Currently that eclass does not support multiple snapshot
> sources/tarballs.
Use case?
--
Best regards,
Michał Górny
signature.asc
Description: PGP signature
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 11:02 AM, Dirkjan Ochtman wrote: > If the tree was bad before you pushed, then it's not your fault the > tree is bad. You're only responsible for the commits you bring into > the tree, so if you're merging contributor's unsigned changesets, you > merge them with a signature of your own. Yup, but the fact that the tree is bad is still a problem, even if it isn't my fault. > If the hacker has unfettered access to the server where the repository > lives, we probably have bigger problems, as they can get whatever > rsynced to all our users. I guess we could have rsync process check > that the cset it's about to push out to mirrors is signed? So, the whole point of signing is that it lets you prove that the repository is uncompromised. If we're going to assume that the server is secure, then we don't need signatures - whatever is on the server is by definition correct. A robust security infrastructure is already spelled out in a GLEP (though that one is dated). Ideally it should be verifiable from end to end, so that when you run emerge if a package has been tampered with it will just refuse to install it. Since we don't distribute the whole git repository the git commits only get us part of the way there. However, if every step of the distribution assumes that the previous step could have been compromised that would be a good start. Again, we don't need to be there 100% to go live. However, I think that was the whole point of signing commits. If we aren't going to add any assurance at all with our signing practices, then there isn't much point in having them. Rich
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 6:06 PM, Rich Freeman wrote: > Again, we don't need to be there 100% to go live. However, I think > that was the whole point of signing commits. If we aren't going to > add any assurance at all with our signing practices, then there isn't > much point in having them. True. However, I still think my idea of security (the tip of tree must always be signed by a gentoo.org committer) and your idea of security (every cset must be signed by a gentoo.org committer) give similar security guarantees in the end. Any user will rely on the last committer to have faithfully signed for an uncompromised tree. Any committer will rely on the previous committer to have faithfully signed for an uncompromised tree. So to prevent your scenario, we'd have to get everyone to check the signature of the tip of tree they pulled before committing/merging. Having every cset signed is something that might make verification slightly easier, but having all previous tips signed (i.e. merges) should be sufficient (if we can rely on committers to review changesets from other committers they pull from). Cheers, Dirkjan
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 12:19 PM, Dirkjan Ochtman wrote: > So to prevent your scenario, we'd > have to get everyone to check the signature of the tip of tree they > pulled before committing/merging. How can we be sure this has happened? This is the problem with signed manifests today. I can sign a manifest, but I didn't actually check all the files inside it, and the file might or might not have been signed before I modified it, and most likely I didn't even check the signature even if it was there. Anything we do has to be automated to be of any real value. Ideally if something goes wrong it should be as detectable as possible. Warts and all the current system hasn't broken down yet. However, if we ever did find out about an intrusion in our cvs repository, we'd essentially have to do a 100% code review to be sure it was OK, and that includes checking all tarballs on mirrors. With signed commits we could verify that the tree was intact, and if anything bad was found we could pinpoint exactly whose key was compromised and do a focused check on their commits. Rich
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 7:25 PM, Rich Freeman wrote: > Anything we do has to be automated to be of any real value. Ideally > if something goes wrong it should be as detectable as possible. Yeah, but you'd have to part of that at every developer's box. Can we just agree that having the tip of the main tree always signed will be enough for now, and postpone the rest of the discussion until later? Cheers, Dirkjan
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 04, 2012 at 08:45:42PM +0200, Dirkjan Ochtman wrote: > On Mon, Jun 4, 2012 at 7:25 PM, Rich Freeman wrote: > > Anything we do has to be automated to be of any real value. ??Ideally > > if something goes wrong it should be as detectable as possible. > > Yeah, but you'd have to part of that at every developer's box. > > Can we just agree that having the tip of the main tree always signed > will be enough for now, and postpone the rest of the discussion until > later? ToT is always going to be signed. If it *isn't* signed, either the infra machinery is broken and not rejecting commits that it should reject, or someone is trojaning the repo (either via an infra compromise, local compromise, or via man in the middle). One thing people need to keep in mind here is that when you sign the commit, you're signing off on the history implicitly. Directly addressing freeman's comment about "people sign the manifest but don't look at what they're signing", when it comes to git signage, bluntly, people doing that shouldn't have access- if they can't be arsed to validate what they're signing, then trusting them w/ the tree is probably questionable. Harsh, but frankly, sane people don't sign enforcable contracts w/out verifying what they're signing (note the 'enforcable' bit, stated to head off the EULA rathole discussion); this isn't any different frankly. ~harring
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 3:10 PM, Brian Harring wrote: > One thing people need to keep in mind here is that when you sign the > commit, you're signing off on the history implicitly. Directly > addressing freeman's comment about "people sign the manifest but don't > look at what they're signing", when it comes to git signage, bluntly, > people doing that shouldn't have access- if they can't be arsed to > validate what they're signing, then trusting them w/ the tree is > probably questionable. I suspect that you're missing my point. The argument was made that as long as merge commits are signed you know that unsigned commits referenced by them are OK. However, some of those commits might have been already in gentoo-x86 and I doubt that anybody is going to check those. If I have a perfect commit, I do a git pull and a git push and the result is a merge that references whatever was in gentoo-x86 before, whether placed there by dev, or hacker, or whatever. Unless I go back and review the existing gentoo-x86 history (and likely have to repeat the process when somebody else does a push before I do), I can't vouch for what was in there already - just what I'm adding. The reason I mentioned maifests is that they have the same issue. If I keyword an arch on foo-1.4.5, I sign the manifest. That doesn't mean that I checked every file in the package's directory tree for issues. At most I checked foo-1.4.5, but I can't sign off on just 1.4.5 - I have to sign off on everything. Also, when I sign off on 1.4.5, I'm really just signing off for the keyword change, not the piece of buggy code I didn't write on line 37 of the ebuild. Of course when merging a pile of commits into the tree you should check all of them to make sure they're fine (or rather that the end result of them is fine - no absolute need to squash together bug introductions and fixes even if that is nicer). However, I'm not sure I'd extend that to checking commits ALREADY in gentoo-x86 made by some other dev. The general principle is that if you change something in the tree, you should be responsible for what you changed, and that makes sense. Rich
Re: [gentoo-dev] [PATCH vcs-snapshot] Use ${WORKDIR}/${P} rather than ${S} to support ${S} overrides.
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 06/04/2012 05:50 PM, Michał Górny wrote:
> On Mon, 04 Jun 2012 16:20:02 +0200 hasufell
> wrote:
>
>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA1
>>
>> On 06/04/2012 11:59 AM, Michał Górny wrote:
>>> One could set S to work on a subtree of the tarball rather than
>>> the whole tarball. Considering that, it's probably better to
>>> use ${WORKDIR}/${P} rather than ${S}.
>>>
>>> Fixes: https://bugs.gentoo.org/show_bug.cgi?id=419479 ---
>>> gx86/eclass/vcs-snapshot.eclass |5 +++-- 1 file changed, 3
>>> insertions(+), 2 deletions(-)
>>>
>>> diff --git a/gx86/eclass/vcs-snapshot.eclass
>>> b/gx86/eclass/vcs-snapshot.eclass index 6748360..23cd696 100644
>>> --- a/gx86/eclass/vcs-snapshot.eclass +++
>>> b/gx86/eclass/vcs-snapshot.eclass @@ -9,7 +9,8 @@ #
>>> @DESCRIPTION: # This eclass provides a convenience src_unpack()
>>> which does support # working with snapshots generated by
>>> various VCS-es. It unpacks those -# to ${S} rather than the
>>> original directory containing commit id. +# to ${WORKDIR}/${P}
>>> rather than the original directory containing +# commit id. # #
>>> Note that this eclass handles only unpacking. You need to
>>> specify # SRC_URI yourself, and call any autoreconfiguration as
>>> necessary. @@ -41,5 +42,5 @@ vcs-snapshot_src_unpack() {
>>>
>>> # github, bitbucket: username-projectname-hash # gitweb:
>>> projectname-tagname-hash - mv *-*-[0-9a-f]*[0-9a-f]/ "${S}" ||
>>> die + mv *-*-[0-9a-f]*[0-9a-f]/ "${WORKDIR}"/${P} || die }
>>
>> Currently that eclass does not support multiple snapshot
>> sources/tarballs.
>
> Use case?
>
ANY case where I need more than one tarball/source. There are just not
many because this eclass is widely unknown.
But minetest in sunrise for example which has two different repos, one
for the engine, one for the data. It's currently split in two, but I
guess I will merge those soon.
Lately there was an ebuild proposal in sunrise too which had that
issue, see here https://gist.github.com/2829184
It would also enable me to use gtk-youtube-viewer and youtube-viewer
in one ebuild with vcs-snapshot eclass while adding a gtk useflag
(currently split too).
Otherwise I will have to fix it on my own again.
I am pretty sure it will get more in the future when people start to
use this eclass or it will just prevent them from using it.
I find the logic very clear:
SRC="https://my/github/shit -> ${P}.tar.gz"
results in ${WORKDIR}/${P}
and
SRC="https://my/github/shit -> ${P}-src.tar.gz"
results in ${WORKDIR}/${P}-src
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJPzQvIAAoJEFpvPKfnPDWzOikH/0R35jeWyWj1j7RNxQ9Rbpt6
JPSBmNx2ZOdFDF8jIqbOERwFwSe87mLitOrZafCxNL4eNZmhP4lXTcVny574CxuS
DE6k5n7k265eYzZm4A/1CjdKp4ptTzpGVil4/iVObFaz3B7tS5PCvqzySWc/ZO/7
040x2d4g94nqkcMM130ua970pbgO7MWZsVsBH3mnLE4AGCZY4vtRlH9dMOHbcAgG
46Vuu9/m4BfGadfGqNVTZz7KbxDBt5h3vOHZZe6JEf3smMr9GiK/gcIc4l97faN7
B601V8kIBWQhPENgeh+SZsMpx24E8DIMvLA7ltMLF8/WLULQBxzW3xR4/CxoNOk=
=aaKu
-END PGP SIGNATURE-
Re: [gentoo-dev] [PATCH vcs-snapshot] Use ${WORKDIR}/${P} rather than ${S} to support ${S} overrides.
On Mon, 04 Jun 2012 21:26:00 +0200
hasufell wrote:
> But minetest in sunrise for example which has two different repos, one
> for the engine, one for the data. It's currently split in two, but I
> guess I will merge those soon.
Why? Is there a good reason to merge two repos into one ebuild? Does
upstream guarantee that the releases will always be synced? Does it
benefit users?
> Lately there was an ebuild proposal in sunrise too which had that
> issue, see here https://gist.github.com/2829184
That's a more likely case. But still such a change would involve
changing an established API heavily which I really dislike.
> It would also enable me to use gtk-youtube-viewer and youtube-viewer
> in one ebuild with vcs-snapshot eclass while adding a gtk useflag
> (currently split too).
> Otherwise I will have to fix it on my own again.
Once again: does it benefit user? Or just does it imply that starting
or stopping to use gtk part requires user to rebuild the whole thing?
> I find the logic very clear:
>
> SRC="https://my/github/shit -> ${P}.tar.gz"
> results in ${WORKDIR}/${P}
> and
> SRC="https://my/github/shit -> ${P}-src.tar.gz"
> results in ${WORKDIR}/${P}-src
I really don't mind the logic. I'm just aware that it is a little late
to introduce such a destructive change, especially that you yourself
mentioned that it will break existing ebuilds.
I will be happy to implement it if you can get more approval for that
change. Or else we should consider jumping with the eclass to -r1 while
it isn't widespread too much.
--
Best regards,
Michał Górny
signature.asc
Description: PGP signature
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 04, 2012 at 03:27:03PM -0400, Rich Freeman wrote: > On Mon, Jun 4, 2012 at 3:10 PM, Brian Harring wrote: > > One thing people need to keep in mind here is that when you sign the > > commit, you're signing off on the history implicitly. ?Directly > > addressing freeman's comment about "people sign the manifest but don't > > look at what they're signing", when it comes to git signage, bluntly, > > people doing that shouldn't have access- if they can't be arsed to > > validate what they're signing, then trusting them w/ the tree is > > probably questionable. > > I suspect that you're missing my point. The argument was made that as > long as merge commits are signed you know that unsigned commits > referenced by them are OK. However, some of those commits might have > been already in gentoo-x86 and I doubt that anybody is going to check > those. Going over the details on the offchance you're misunderstanding part of it: A signed commit is a signing of the git metadata; tree hash (literally, the state of the tree), committer, author, message, and parent sha1. Each git commit includes it's parent sha1 in it; this gives a locked history for a given commit sha1 (unless someone preimages sha1). What matters is that the leaf node, the final point in the graph, is signed- that's a dev sign off on effectively that they created that particular locked history. Realistically signing of each node is preferable, but the leaf is the minimal required. The dev, prior to signing that, should be verifying what they're adding (moreso, what exists between last signed rev and theirs), they agree to and know of. Specifically, they're asserting their addition. When the dev resync's, part of the process *should* be validating that the new remote ToT is actually signed by a trusted key; aka, another dev. For merge commits, the same applies here- due to the rather nice sha1 hack of git, signing the merge commit has the same angle, the dev signs off on the new addition effectively (for merge commits, moreso that the resultant integration is what they intended). Now to pick at your statement; "the dev is signing off on commits already in x86"; think through this scheme; each dev signs off as they go on their specific changes, validating backwards is doable- meaning you can validate that all commits in the vcs were signed off by trusted folk (ie, dev's). Your argument is either based on misunderstanding how this works (potentially just ignoring the pull validation devs have to do), or assuming that the dev is responsible for signing off on *everything* in the tree's proceeding history just because they're signing a specific history/tree sha. As indicated above, they're signing off essentially on each gap between signing, the delta. > If I have a perfect commit, I do a git pull and a git push and > the result is a merge that references whatever was in gentoo-x86 > before, whether placed there by dev, or hacker, or whatever. Unless I > go back and review the existing gentoo-x86 history (and likely have to > repeat the process when somebody else does a push before I do), I > can't vouch for what was in there already - just what I'm adding. For someone to inject something into the history requires basically a sha1 preimage attack, or the dev to be hit by an injection attack, along w/ the dev *skipping* validation of the pull to current ToT. If we require every commit to be signed, that blocks the potential for a careless dev skipping the valdation to sign off on things that were injected. This is a bit honerous, and is why signed pushes were the preference (and is a bit of an abuse what we're doing now). > The reason I mentioned maifests is that they have the same issue. If > I keyword an arch on foo-1.4.5, I sign the manifest. That doesn't > mean that I checked every file in the package's directory tree for > issues. At most I checked foo-1.4.5, but I can't sign off on just > 1.4.5 - I have to sign off on everything. Also, when I sign off on > 1.4.5, I'm really just signing off for the keyword change, not the > piece of buggy code I didn't write on line 37 of the ebuild. Manifests aren't particularly comparable, nor is it worth trying to do so. Technically w/ CVS manifests you can isolate down just the delta, and infer that the signage was for that, but the tools don't particularly exist for it, nor is it worth it; cvs just isn't designed sanely to allow for this. With git however, since the signing can be integrated directly into the revs themselves, you *are* directly signing the deltas; as said, w/ cvs technically you are (even if you're signing all of the results). An additional note btw; to be clear, the only manifest file that exists in a git tree is a thin manifest holding checksums for the distilfes involved. If you're thinking of rsync distributed trees (which aren't thin manifests), the plan was to enhance the existing rsync snapshot generation sc
Re: [gentoo-dev] [PATCH vcs-snapshot] Use ${WORKDIR}/${P} rather than ${S} to support ${S} overrides.
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 06/04/2012 10:06 PM, Michał Górny wrote:
> On Mon, 04 Jun 2012 21:26:00 +0200 hasufell
> wrote:
>
>> But minetest in sunrise for example which has two different
>> repos, one for the engine, one for the data. It's currently split
>> in two, but I guess I will merge those soon.
>
> Why? Is there a good reason to merge two repos into one ebuild?
> Does upstream guarantee that the releases will always be synced?
> Does it benefit users?
In this case yes. They are released with the exact same tags as you
can see in those ebuilds.
>
>> It would also enable me to use gtk-youtube-viewer and
>> youtube-viewer in one ebuild with vcs-snapshot eclass while
>> adding a gtk useflag (currently split too). Otherwise I will have
>> to fix it on my own again.
>
> Once again: does it benefit user? Or just does it imply that
> starting or stopping to use gtk part requires user to rebuild the
> whole thing?
Eclasses do not benefit any user. They benefit developers. I would
simply do similar stuff on my own in the ebuild instead of using
vcs-snapshot eclass then.
>
>> I find the logic very clear:
>>
>> SRC="https://my/github/shit -> ${P}.tar.gz" results in
>> ${WORKDIR}/${P} and SRC="https://my/github/shit ->
>> ${P}-src.tar.gz" results in ${WORKDIR}/${P}-src
>
> I really don't mind the logic. I'm just aware that it is a little
> late to introduce such a destructive change, especially that you
> yourself mentioned that it will break existing ebuilds.
So? We fix it.
>
> I will be happy to implement it if you can get more approval for
> that change. Or else we should consider jumping with the eclass to
> -r1 while it isn't widespread too much.
>
I don't see the point in bumping it, because it's not widespread.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJPzR7lAAoJEFpvPKfnPDWz5W8H/0Je1mE/Vo7X+46TpuZZyi/3
RJaJMYETeQbbhPM6ACIXtHk629fGCz9Oda7J0YG4LMCYTbxU5MNElZSjbV4aThYD
MkSoQlSw/RIBuSEaffWRkAtbmNovHzd+nUyK8cHJTYXffi4CmClPXPPTqGAaRbC/
yJf6JBEfMLK/6ps10eMwf7D/m5ZJUYIPJ1m7DmlUqjpr8R8v2bVbjqB//M9ig7KO
yl/W5qzlBa2UAw/Gjgi0ITdDKs5sem7J8+PbVZKED5K0sD10YxZKMImCymJSlFkR
gzqZi99qdAs8uhZ1K6h8ozkBLglxkT54IZ8Kn3LWwiQ0/I2xRNgX8Ugt1EQnrQM=
=X+fU
-END PGP SIGNATURE-
Re: Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
> A signed commit is a signing of the git metadata; tree hash > (literally, the state of the tree), committer, author, message, and > parent sha1. Each git commit includes it's parent sha1 in it; this > gives a locked history for a given commit sha1 (unless someone > preimages sha1). What matters is that the leaf node, the final point > in the graph, is signed- that's a dev sign off on effectively that > they created that particular locked history. Realistically signing of > each node is preferable, but the leaf is the minimal required. No. What is signed is the "new data" plus the parent hash(es). No such thing as a "tree hash". -- Andreas K. Huettel Gentoo Linux developer kde, sci, arm, tex, printing signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 4:41 PM, Brian Harring wrote: > > If that doesn't answer your question/concerns, be more explicit > please. How about a scenario: 1. Gentoo dev commits a bunch of stuff to the tree. Top of tree is signed. 2. Hacker commits something to the tree. Top of tree is not signed. No need for preimage attacks or whatever on sha1 - they just log into the server and do a git commit or whatever right into the tree. 3. Gentoo dev commits a bunch of stuff to the tree. Top of tree is signed. 4. Rsync mirror update happens - top of tree is signed, so update proceeds normally. If you go back and look at the tree you see a bunch of signed and unsigned commits. How do you easily detect how the unsigned ones got there (via a dev with a merge commit, or via other means)? Either way they'll be parents of merge commits - since merge commits have two parents - the pre-commit gentoo-x86 tree, and the incoming commits. Andreas - I'm pretty sure a merge commit still includes a tree. Rich
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, 04 Jun 2012 22:52:25 +0200 "Andreas K. Huettel" wrote: > No. What is signed is the "new data" plus the parent hash(es). > > No such thing as a "tree hash". http://eagain.net/articles/git-for-computer-scientists/ Might clear things up a bit. -- Ciaran McCreesh signature.asc Description: PGP signature
[gentoo-dev] [gentoo-portage-dev] About forcing rebuilds of other packages issue
Hello, will send this to gentoo-dev mailing list per Zac's suggestion ;): Probably Zac already remembers my suggestion of: https://bugs.gentoo.org/show_bug.cgi?id=413619 Sorry for insisting a bit on it but this issue bites me periodically. Months ago, I was able to administrate myself some of my father and uncles systems in their jobs and homes but, since I moved to Madrid this year, I am not able to administrate them directly. They usually do a good job maintaining them, the only issue I see they hit from time to time is forgetting to run JUST AFTER updating their systems revdep-rebuild (well, this is so common that they usually don't forget to), rebuild dbus-glib/gobject-introspection after major glib update, rebuild X11 drivers... This is because, even if all this information is recorded in /var/log/portage/elog/summary.log, currently, that log file is cluttered of a lot of other elog lines that are not related at all with this important task of rebuilding packages. This is why I suggested: https://bugs.gentoo.org/show_bug.cgi?id=413619 That would create a new "erebuild" (or whatever the name you prefer) to ONLY contain exact command to run by admin to have a safe system after update. It would have as main advantage: - Looks easier to implement. - It relies in current and existing tools (python-updater, perl-cleaner, "q", equery...), then, they could be used just now via a script running all of them. - It also looks much more "professional" to try to unify a bit what commands to run ;) (currently, some ebuilds tells you to manually re-emerge packages and some people wrongly run "emerge dbus-glib" when they should run "emerge -1 dbus-glib". Telling us to people what exact command they need to copy&paste&run will help to get their systems cleaner also. Zac kindly pointed me to: https://bugs.gentoo.org/show_bug.cgi?id=192319 The problem of that one is that, even if it would be "the perfect solution": - Looks to be stalled for a long time. - Looks to need a lot of functions (like revdep-rebuild, python-updater...) to be merged in portage itself. It will then probably take a lot of time to get them integrated (specially seeing we are still not able to use preserve-libs because it looks to cause some other problems) - In that bug report I have also seen discussion about whether handle this only via SLOTs (that personally think it will be even harder to achieve for all packages in the tree showing this kind of problems when updating, for example, I doubt how "glib" - "dbus-glib/g-i" case could be handled in this way. - Looks like there is no consensus about what to do and, then, this could probably be implemented on eapi... 7? While former could probably be implemented much sooner (probably even in eapi5) This is why I think we should try to push a bit my first suggestion for the short term until "the perfect one" is ready as, until then, we are having for years a problem that, personally, I think it should be handled a bit better. Thanks a lot for your attention signature.asc Description: This is a digitally signed message part
[gentoo-dev] About forcing rebuilds of other packages issue
Hello, will send this to gentoo-dev mailing list per Zac's suggestion ;): Probably Zac already remembers my suggestion of: https://bugs.gentoo.org/show_bug.cgi?id=413619 Sorry for insisting a bit on it but this issue bites me periodically. Months ago, I was able to administrate myself some of my father and uncles systems in their jobs and homes but, since I moved to Madrid this year, I am not able to administrate them directly. They usually do a good job maintaining them, the only issue I see they hit from time to time is forgetting to run JUST AFTER updating their systems revdep-rebuild (well, this is so common that they usually don't forget to), rebuild dbus-glib/gobject-introspection after major glib update, rebuild X11 drivers... This is because, even if all this information is recorded in /var/log/portage/elog/summary.log, currently, that log file is cluttered of a lot of other elog lines that are not related at all with this important task of rebuilding packages. This is why I suggested: https://bugs.gentoo.org/show_bug.cgi?id=413619 That would create a new "erebuild" (or whatever the name you prefer) to ONLY contain exact command to run by admin to have a safe system after update. It would have as main advantage: - Looks easier to implement. - It relies in current and existing tools (python-updater, perl-cleaner, "q", equery...), then, they could be used just now via a script running all of them. - It also looks much more "professional" to try to unify a bit what commands to run ;) (currently, some ebuilds tells you to manually re-emerge packages and some people wrongly run "emerge dbus-glib" when they should run "emerge -1 dbus-glib". Telling us to people what exact command they need to copy&paste&run will help to get their systems cleaner also. Zac kindly pointed me to: https://bugs.gentoo.org/show_bug.cgi?id=192319 The problem of that one is that, even if it would be "the perfect solution": - Looks to be stalled for a long time. - Looks to need a lot of functions (like revdep-rebuild, python-updater...) to be merged in portage itself. It will then probably take a lot of time to get them integrated (specially seeing we are still not able to use preserve-libs because it looks to cause some other problems) - In that bug report I have also seen discussion about whether handle this only via SLOTs (that personally think it will be even harder to achieve for all packages in the tree showing this kind of problems when updating, for example, I doubt how "glib" - "dbus-glib/g-i" case could be handled in this way. - Looks like there is no consensus about what to do and, then, this could probably be implemented on eapi... 7? While former could probably be implemented much sooner (probably even in eapi5) This is why I think we should try to push a bit my first suggestion for the short term until "the perfect one" is ready as, until then, we are having for years a problem that, personally, I think it should be handled a bit better. Thanks a lot for your attention signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Git braindump: 2 of N: developer interaction (merge co-ordinators)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 06/04/2012 03:25 PM, Brian Harring wrote: > While I do grok the potential issue of someone being a hog > (specifically via blasting commit by commit rather than building up > work locally, then pushing it in chunks), frankly... I'm not that > concerned about it, and would rather deal w/ it if/when it occurs. > The nature of our commits for the most part are standalone from > others- that's not true of the kernel/mozilla, thus why I don't > think their issues are necessarily ours. True. We already have maintainers and herds as responsible (sole editors) entities for locations (packages). But, we have arch teams editing ebuild/KEYWORDS, which alters Manifest/EBUILD lines. Resulting in potential clashes (not fast-forwardable), if the herd or maintainer does bumps or cleanups. Will these Manifest lines (and the arch team inflicted Manifest changes)? And we have orphaned (maintainer-needed) and "everyone can fix it" herds like desktop-*). This results in a large group of potential bug-fixers (committers) and the potential of concurrent edits. This can be managed by using bugzilla IN_PROGRESS as a lock state, but I thats not very practicable/needs disciplin/is annoying. But this is no regression compared to CVS, we just need to signal clashed. Last assumed hot spot imho is package.mask with ~700 commits in the last 4.5 months (one every 4.6 hours) and ~620 commits in **/package.use.mask. Not that much. According to robbat2 data (gentoo-commit tarball) we have ~400k commits in gentoo-x86 (w/o proj,xml) in 4.7 years, that's 6.2 per hour averaged. But I've to look into the data to see trends (# developers, daylight). Michael - -- Gentoo Dev http://xmw.de/ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAk/NOFQACgkQknrdDGLu8JARlwEAldk7GAEqCrd5mSsDgC69e5uQ aqivvwbDpNWSgfwJqwUA/jjlByEncXXPVia11BALSdDf1elrL/3qAf5ktCtxx/m0 =pJFc -END PGP SIGNATURE-
Re: [gentoo-dev] Git braindump: 2 of N: developer interaction (merge co-ordinators)
On Tue, Jun 05, 2012 at 12:36:04AM +0200, Michael Weber wrote: > On 06/04/2012 03:25 PM, Brian Harring wrote: > > While I do grok the potential issue of someone being a hog > > (specifically via blasting commit by commit rather than building up > > work locally, then pushing it in chunks), frankly... I'm not that > > concerned about it, and would rather deal w/ it if/when it occurs. > > The nature of our commits for the most part are standalone from > > others- that's not true of the kernel/mozilla, thus why I don't > > think their issues are necessarily ours. > True. > > We already have maintainers and herds as responsible (sole editors) > entities for locations (packages). > > But, we have arch teams editing ebuild/KEYWORDS, which alters > Manifest/EBUILD lines. Resulting in potential clashes (not > fast-forwardable), if the herd or maintainer does bumps or cleanups. > > Will these Manifest lines (and the arch team inflicted Manifest changes)? Converting to git, we'll switch over to thin manifests- they're *just* the checksums for the distfiles, no need for the rest since git already provides that verification implicitly. That just leaves conflict w/in ebuilds, which is a valid "the dev needs to deal with this themselves" scenario imo. > According to robbat2 data (gentoo-commit tarball) we have ~400k > commits in gentoo-x86 (w/o proj,xml) in 4.7 years, that's 6.2 per hour > averaged. > But I've to look into the data to see trends (# developers, daylight). One thing to note- that's *individual* commits, and probably a mildly jacked up number due to the double tap requirement of commiting manifests to CVS. What I'm driving at is that there's a difference between commits/revisions, and pushs; I expect our push rate to be less; I'd be surprised if we're doing 1:1 commit/push rate. The conflict rate should be less painful for people in that light, or at least has been in my experience thus far. Btw, good catch on package.mask. Hhadn't thought of that, that *will* be the most contentious point. That can be dealt w/ via having git on portage-1 profile format so we'd have package.mask as directories (which Ciaran will validly hate, and I won't like due to having to write the portage-1 -> PMS translater for rsync distribution), or coming up w/ a different way to split the commits across multiple files, rather than a single. That's assuming package.mask becomes a significant conflict point also. Frankly I'd rather deal w/ that problem when it arrises, rather than trying to optimize for it now. ~harring
Re: [gentoo-dev] Re: metadata/md5-cache
On Sun, Jun 03, 2012 at 09:25:43AM +, Robin H. Johnson wrote: > On Sun, Jun 03, 2012 at 08:31:43AM +, Duncan wrote: > > Micha?? G??rny posted on Sun, 03 Jun 2012 09:22:04 +0200 as excerpted: > > > > >> Even if only the files metatdata changes, that still adds a significant > > >> cost to an rsync. > > > I wonder when it will come to the point where git will be more efficient > > > than rsync. Or maybe it would be already? > > Handwavey guess, but I've figured git to be more efficient client-side > > for some time. Server-side I don't know about, but I've presumed that's > > the reason the switch-to-git plans haven't included switching the default > > for user-syncs to git. I expect user/client side, git would be more > > efficient already, but as I said, that's handwavey guesses. > No, the switch to git will NOT help users, it isn't more efficient. > > They will still be best served by rsync, for a couple of reasons: > 1. metadata cache is NOT available in Git. Sidenote, and this is mildly insane, I'd thought about submodules for this; basically every rsync window, we dump the metadata into vcs, which devs can pull down and make use of. I've also not experimented w/ this workflow, so it could be batshit insane. Anyone game to experiment? ~harring
Re: [gentoo-dev] About forcing rebuilds of other packages issue
On 06/04/2012 02:29 PM, Pacho Ramos wrote: > - Looks like there is no consensus about what to do and, then, this > could probably be implemented on eapi... 7? While former could probably > be implemented much sooner (probably even in eapi5) Ciaran has been advocating "SLOT operator" dependencies for this, but those are not designed to work with unslotted packages, which leads to the issues discussed in bug 414955: https://bugs.gentoo.org/show_bug.cgi?id=414955#c10 -- Thanks, Zac
Re: [gentoo-dev] Git braindump: 2 of N: developer interaction (merge co-ordinators)
Brian Harring wrote: > rather deal w/ that problem when it arrises, rather than trying to > optimize for it now. I'm strongly in favor of ripping off the bandaid, fast and hard! Some hooks to block problematic pushes, and let people learn as they go. It will take weeks to months for everyone to migrate, if not their repos then at least their mindset. No amount of preparation will reduce that significantly for all developers, so too much preparation is just wasted effort. //Peter
[gentoo-dev] Re: [PATCH eutils] Move remove_libtool_files() from autotools-utils for wider use.
On Mon, 28 May 2012 09:58:56 +0200 Michał Górny wrote: > As autotools-utils exports phase functions, it will be better if > remove_libtool_files() functions would be somewhere else. Thank you. -- fonts, gcc-porting toolchain, wxwidgets @ gentoo.org signature.asc Description: PGP signature
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, Jun 4, 2012 at 10:41 PM, Brian Harring wrote: > The dev, prior to signing that, should be verifying what they're > adding (moreso, what exists between last signed rev and theirs), they > agree to and know of. Specifically, they're asserting their addition. What Rich is arguing (and which I think makes some sense) is that people will probably not be inclined to verify the signature of the tree they just pulled from gentoo-x86. We can't really force them too, since it happens on their own machine. Still, I think we should drop this discussion for now. Cheers, Dirkjan
Re: [gentoo-dev] [PATCH eutils] Introduce prune_libtool_files().
On Thursday 31 May 2012 08:55:25 Michał Górny wrote:
> +# Note: this function implicitly calls pkg-config. You should add it to
> +# your DEPEND when using it.
should clarify: implicitly calls pkg-config when your package provides a .pc.
> + if [[ ! ${removing_all} ]]; then
> + local f
> + local tf=${T}/prune-lt-files.pc
> + local pkgconf=$(tc-getPKG_CONFIG)
> +
> + while IFS= read -r -d '' f; do # for all .pc files
> + sed -e '/^Requires:/d' "${f}" > "${tf}"
> + for arg in $("${pkgconf}" --libs "${tf}"); do
missing `local arg`
> + done < <(find "${D}" -type f -name '*.pc' -print0)
not a big deal since it'll get cleaned anyways, but this doesn't clean up $tf
when done
if no one else has feedback, i guess merge it
-mike
signature.asc
Description: This is a digitally signed message part.
Re: [gentoo-dev] multiprocessing.eclass: doing parallel work in bash
On Sunday 03 June 2012 18:16:30 Zac Medico wrote:
> On 06/02/2012 10:08 PM, Mike Frysinger wrote:
> > # @FUNCTION: _multijob_fork # @INTERNAL # @DESCRIPTION: # Do the
> > actual book keeping. _multijob_fork() { [[ $# -eq 1 ]] || die
> > "incorrect number of arguments"
> >
> > local ret=0 [[ $1 == "pre" ]] && : $(( ++mj_num_jobs ))
> > if [[ ${mj_num_jobs} -ge ${mj_max_jobs} ]] ; then
> > multijob_finish_one
> > ret=$?
> > fi
> > [[ $1 == "post" ]] && : $(( ++mj_num_jobs ))
> > return ${ret}
>
> The "pre" logic seems wrong. Consider an initial state of
> mj_num_jobs=0 and mj_max_jobs=1. It will increment mj_num_jobs to 1,
> so [[ 1 -ge 1 ]] is true, and then call multijob_finish_one even
> though no jobs have started yet? Wouldn't that deadlock
> multijob_finish_one, as it waits for a reply from a job that doesn't
> exist yet?
yes, i inverted the cases in this func
-mike
signature.asc
Description: This is a digitally signed message part.
Re: [gentoo-dev] multiprocessing.eclass: doing parallel work in bash
v4
-mike
# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
# @ECLASS: multiprocessing.eclass
# @MAINTAINER:
# base-sys...@gentoo.org
# @AUTHOR:
# Brian Harring
# Mike Frysinger
# @BLURB: parallelization with bash (wtf?)
# @DESCRIPTION:
# The multiprocessing eclass contains a suite of functions that allow ebuilds
# to quickly run things in parallel using shell code.
#
# It has two modes: pre-fork and post-fork. If you don't want to dive into any
# more nuts & bolts, just use the pre-fork mode. For main threads that mostly
# spawn children and then wait for them to finish, use the pre-fork mode. For
# main threads that do a bit of processing themselves, use the post-fork mode.
# You may mix & match them for longer computation loops.
# @EXAMPLE:
#
# @CODE
# # First initialize things:
# multijob_init
#
# # Then hash a bunch of files in parallel:
# for n in {0..20} ; do
# multijob_child_init md5sum data.${n} > data.${n}
# done
#
# # Then wait for all the children to finish:
# multijob_finish
# @CODE
if [[ ${___ECLASS_ONCE_MULTIPROCESSING} != "recur -_+^+_- spank" ]] ; then
___ECLASS_ONCE_MULTIPROCESSING="recur -_+^+_- spank"
# @FUNCTION: makeopts_jobs
# @USAGE: [${MAKEOPTS}]
# @DESCRIPTION:
# Searches the arguments (defaults to ${MAKEOPTS}) and extracts the jobs number
# specified therein. Useful for running non-make tools in parallel too.
# i.e. if the user has MAKEOPTS=-j9, this will echo "9" -- we can't return the
# number as bash normalizes it to [0, 255]. If the flags haven't specified a
# -j flag, then "1" is shown as that is the default `make` uses. Since there's
# no way to represent infinity, we return 999 if the user has -j without a
number.
makeopts_jobs() {
[[ $# -eq 0 ]] && set -- ${MAKEOPTS}
# This assumes the first .* will be more greedy than the second .*
# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
local jobs=$(echo " $* " | sed -r -n \
-e
's:.*[[:space:]](-j|--jobs[=[:space:]])[[:space:]]*([0-9]+).*:\2:p' \
-e 's:.*[[:space:]](-j|--jobs)[[:space:]].*:999:p')
echo ${jobs:-1}
}
# @FUNCTION: multijob_init
# @USAGE: [${MAKEOPTS}]
# @DESCRIPTION:
# Setup the environment for executing code in parallel.
# You must call this before any other multijob function.
multijob_init() {
# When something goes wrong, try to wait for all the children so we
# don't leave any zombies around.
has wait ${EBUILD_DEATH_HOOKS} || EBUILD_DEATH_HOOKS+=" wait"
# Setup a pipe for children to write their pids to when they finish.
local pipe="${T}/multijob.pipe"
mkfifo "${pipe}"
redirect_alloc_fd mj_control_fd "${pipe}"
rm -f "${pipe}"
# See how many children we can fork based on the user's settings.
mj_max_jobs=$(makeopts_jobs "$@")
mj_num_jobs=0
}
# @FUNCTION: multijob_child_init
# @USAGE: [--pre|--post] [command to run in background]
# @DESCRIPTION:
# This function has two forms. You can use it to execute a simple command
# in the background (and it takes care of everything else), or you must
# call this first thing in your forked child process.
#
# The --pre/--post options allow you to select the child generation mode.
#
# @CODE
# # 1st form: pass the command line as arguments:
# multijob_child_init ls /dev
# # Or if you want to use pre/post fork modes:
# multijob_child_init --pre ls /dev
# multijob_child_init --post ls /dev
#
# # 2nd form: execute multiple stuff in the background (post fork):
# (
# multijob_child_init
# out=`ls`
# if echo "${out}" | grep foo ; then
# echo "YEAH"
# fi
# ) &
# multijob_post_fork
#
# # 2nd form: execute multiple stuff in the background (pre fork):
# multijob_pre_fork
# (
# multijob_child_init
# out=`ls`
# if echo "${out}" | grep foo ; then
# echo "YEAH"
# fi
# ) &
# @CODE
multijob_child_init() {
local mode="pre"
case $1 in
--pre) mode="pre" ; shift ;;
--post) mode="post"; shift ;;
esac
if [[ $# -eq 0 ]] ; then
trap 'echo ${BASHPID} $? >&'${mj_control_fd} EXIT
trap 'exit 1' INT TERM
else
local ret
[[ ${mode} == "pre" ]] && { multijob_pre_fork; ret=$?; }
( multijob_child_init ; "$@" ) &
[[ ${mode} == "post" ]] && { multijob_post_fork; ret=$?; }
return ${ret}
fi
}
# @FUNCTION: _multijob_fork
# @INTERNAL
# @DESCRIPTION:
# Do the actual book keeping.
_multijob_fork() {
[[ $# -eq 1 ]] || die "incorrect number of arguments"
local ret=0
[[ $1 == "post" ]] && : $(( ++mj_num_jobs ))
if [[ ${mj_num_jobs} -ge ${mj_max_jobs} ]] ; then
multijob_finish_one
ret=$?
fi
[[ $1 == "pre" ]] && : $(( ++mj_num_jobs ))
return ${ret}
}
# @FUNCTION: multijob_pre_fork
# @DES
Re: [gentoo-dev] Git braindump: 1 of N: merging & git signing
On Mon, 4 Jun 2012 16:57:42 -0400 Rich Freeman wrote: > If you go back and look at the tree you see a bunch of signed and > unsigned commits. How do you easily detect how the unsigned ones got > there (via a dev with a merge commit, or via other means)? Well, that's not a very good solution but the server-side hooks could also verify the tree state before applying new commits. -- Best regards, Michał Górny signature.asc Description: PGP signature
