Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)
Dnia 2014-09-14, o godz. 21:30:36 Tim Harder radher...@gentoo.org napisał(a): On 2014-09-14 10:46, Michał Górny wrote: Dnia 2014-09-14, o godz. 15:40:06 Davide Pesavento p...@gentoo.org napisał(a): How long does the md5-cache regeneration process take? Are you sure it will be able to keep up with the rate of pushes to the repo during peak hours? If not, maybe we could use a time-based thing similar to the current cvs-rsync synchronization. This strongly depends on how much data is there to update. A few ebuilds are quite fast, eclass change isn't ;). I was thinking of something along the lines of, in pseudo-code speaking: systemctl restart cache-regen That is, we start the regen on every update. If it finishes in time, it commits the new metadata. If another update occurs during regen, we just restart it to let it catch the new data. Of course, if we can't spare the resources to do intermediate updates, we may as well switch to cron-based update method. I don't see per push metadata regen working entirely well in this case if this is the only way we're generating the metadata cache for users to sync. It's easy to imagine a plausible situation where a widely used eclass change is made followed by commits less than a minute apart (or shorter than however long it would take for metadata regen to occur) for at least 30 minutes (rsync refresh period for most user-facing mirrors) during a time of high activity. For a metadata recheck (that is, egencache run with no changes): a. cold cache ext4: real3m54.321s user0m44.413s sys 0m13.497s b. warm cache ext4: real0m40.672s user0m35.087s sys 0m 4.687s I will try to re-run that on btrfs or reiserfs to get a more meaningful numbers. Now, that results back up your claims. However, if we can get that to 10s, I doubt we would have a major issue. My idea works like this: 1. first update is pushed, 1a. egencache starts rechecking and updating cache, 2. second update is pushed, 2a. previous egencache is terminated, 2b. egencache starts rechecking and updating cache, 2c. egencache finishes in time and commits. The point is, nothing gets committed to the user-reachable location before egencache finishes. And it goes quasi-incrementally, so if another update happens before egencache finished, it only does the 'slow' regen on changed metadata. I will come back with more results soon. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)
Dnia 2014-09-15, o godz. 07:21:35 Patrick Lauer patr...@gentoo.org napisał(a): On Sunday 14 September 2014 15:42:15 hasufell wrote: Patrick Lauer: Are we going to disallow merge commits and ask devs to rebase local changes in order to keep the history clean? Is that going to be sane with our commit frequency? You have to merge or rebase anyway in case of a push conflict, so the only difference is the method and the effect on the history. Currently... CVS allows you to run repoman on an outdated tree and push broken ebuilds with repoman being happy. Git will not allow this. iow, git doesn't allow people to work on more than one item at a time? That'd mean I need half a dozen checkouts just to emulate cvs, which somehow doesn't make much sense to me ... I'd appreciate if you reduced FUD to minimum. What hasufell meant is that you normally don't have three year-old files lying around in checkout because you did 'cvs up -dP' in another directory. With git, you update everything. What you do locally, is totally unrelated. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)
On 14-09-2014 16:56:24 +0200, Michał Górny wrote: Rich Freeman ri...@gentoo.org napisał(a): So, I don't really have a problem with your design. I still question whether we still need to be generating changelogs - they seem incredibly redundant. But, if people really want a redundant copy of the git log, whatever... I don't want them too. However, I'm pretty sure people will bikeshed this to death if we kill them... Especially that rsync has no git log. Not that many users make real use of ChangeLogs, esp. considering how useless messages often are there... Council had some discussions on this topic: http://www.gentoo.org/proj/en/council/meeting-logs/2008-summary.txt http://www.gentoo.org/proj/en/council/meeting-logs/20111011-summary.txt Conclusion back then was that ChangeLog files need to stay. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-dev] Does the scm ebuild masking policy make sense for git?
On Thu, 11 Sep 2014 23:40:32 + hasufell hasuf...@gentoo.org wrote: I don't see [...] It is hard to connect the dots if you don't know about the dots; do your homework to find them, ask questions when you don't. [...] any sense in what you say. You sound confused. Without those dots, your sound of confusion sees no sense.
Re: [gentoo-dev] Does the scm ebuild masking policy make sense for git?
On Sat, 13 Sep 2014 22:44:49 + hasufell hasuf...@gentoo.org wrote: Jauhien Piatlicki: In the ideal country of elves. In the real life it can be not possible to build and install software in a given distribution without downstream patches. You can find examples of such live ebuilds in Gentoo tree. I think it's not appropriate and shouldn't generally be done (with a few exceptions). If the live ebuild needs heavy patching to even work, then don't commit it to the tree. Patches come and go, that has nothing to do with its presence in the Portage tree. We shouldn't remove and add them based on the varying amount of patches, as that is much work with an unfavorable result. Anyway, summarizing, it is completely impossible to be sure that live ebuild will be buildable for you on a given arch in the next 15 min., even if it was so in the last 15 min. That goes for almost all ebuild variables. So you either drop the whole concept of live ebuilds or you do what is reasonable: Given that some users specifically request them, the concept lives on. a) provide consistent ebuild information, including keywords KEYWORDS= is as consistent as it can get. b) ask upstream about their git workflow, which branches they use, what arches they even officially support This is not how we fill in KEYWORDS. c) only add live ebuilds if the upstream git model is something that can be relied upon in one way or another and if you can keep up with the changes The entire package presence depends on this, not just the live ebuilds. If your live ebuild breaks every 15 minutes, then it shouldn't be in the tree. Bleeding edge is designed to break every 15 minutes; live ebuilds are bleeding edge, some users do want bleeding edge in the Portage tree. I actually don't commit live ebuild unless I know that upstream is collaborative and I'v contributed to almost all of the projects which I package as live ebuilds. Consider to step outside the ideal country of elves* and explore. (* quoting Jauhien's words)
Re: [gentoo-dev] RFC: Deprecating and killing the concept of herds
On Thu, 11 Sep 2014 21:00:16 +0100 Markos Chandras hwoar...@gentoo.org wrote: please do not go offtopic discussing the recruitment process. I simply mentioned one of the designated ways we have to ask for help. If you don't like it, propose a better method. Please do not go offtopic about your own getting people on board. I simply suggested improvements to one of the designated ontopic problems that can be seen in herds. If you don't like it, write a better reply.
Re: [gentoo-dev] Does the scm ebuild masking policy make sense for git?
2014-09-10 15:59 GMT+01:00 Tom Wijsman tom...@gentoo.org: On Wed, 10 Sep 2014 13:56:04 + hasufell hasuf...@gentoo.org wrote: Tom Wijsman: On Tue, 09 Sep 2014 19:12:28 + hasufell hasuf...@gentoo.org wrote: Jauhien Piatlicki: When I accept ~arch I expect that no live ebuilds will be built. I think other gentoo users expect the same. Just because users are used to it doesn't make it better. How does it make it better for users that are used to what works well? It improves usability by providing additional information. Usability is not to be found in information that is subject to change. How frequently the list of supported arches does shrink? Is it statistically significant? -- Georg Rudoy
Re: [gentoo-dev] Does the scm ebuild masking policy make sense for git?
2014-09-13 21:03 GMT+01:00 Peter Stuge pe...@stuge.se: I would actually expect there to be a policy which forbids patches on live ebuilds. Make another live ebuild or maybe an overlay if you want to offer a different set of commits than the upstream repo. For me, the whole point of live ebuilds is that they are the latest upstream code, no more, no less. While I agree with the rest of your message, there should be exceptions from this rule IMO. For example, maintainers of my project in other distros keep a set of patches to enable building it with older compilers (as I use C++11 pretty extensively, and gcc 4.7 already cannot swallow all of it). The patches for existing code hardly change ever, probably once in a few months. This is hardly applicable to Gentoo though as corresponding ebuilds already require gcc = 4.8. -- Georg Rudoy
Re: [gentoo-dev] Does the scm ebuild masking policy make sense for git?
On Mon, 15 Sep 2014 10:11:08 +0100 Georg Rudoy 0xd34df...@gmail.com wrote: How frequently the list of supported arches does shrink? Is it statistically significant? The amount of software that exists makes this impossible to determine.
Re: [gentoo-dev] Does the scm ebuild masking policy make sense for git?
2014-09-15 10:24 GMT+01:00 Tom Wijsman tom...@gentoo.org: On Mon, 15 Sep 2014 10:11:08 +0100 Georg Rudoy 0xd34df...@gmail.com wrote: How frequently the list of supported arches does shrink? Is it statistically significant? The amount of software that exists makes this impossible to determine. Let's limit our sample to Gentoo tree then. How frequently arches list shrinked as a result of bumping the version (because the upstream has chosen so, not because of insufficient resources to keep testing all previously stated arches)? -- Georg Rudoy
Re: [gentoo-dev] Does the scm ebuild masking policy make sense for git?
On Mon, 15 Sep 2014 10:28:16 +0100 Georg Rudoy 0xd34df...@gmail.com wrote: Let's limit our sample to Gentoo tree then. How frequently arches list shrinked as a result of bumping the version (because the upstream has chosen so, not because of insufficient resources to keep testing all previously stated arches)? They do not shrink only due to a result of bumping. In the ideal case it is indeed announced and we can do it at bump time; in the less ideal case they rather shrink due to us finding out that a particular major feature does no longer work on an architecture. This could thus also happen due to an incompatibility with one or another dependency for which no further architecture specific code is written, or perhaps due to an architecture specific bug that has arisen. In the ideal case, shrinking wouldn't be a problem; in the less ideal case, having no keywords is preferable to shrinking as continuously testing live ebuilds isn't affordable.
Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)
Hi, On 09/15/2014 01:37 AM, Kent Fredric wrote: On 15 September 2014 11:25, hasufell hasuf...@gentoo.org wrote: Robin said The Git commit-signing design explicitly signs the entire commit, including blob contents, to avoid this security problem. Is this correct or not? I can verify a commit by hand with only the commit object and gpg, but without any of the trees or parents. https://gist.github.com/kentfredric/8448fe55ffab7d314ecb So signing of git commits does not guarantee enough security (taking that SHA1 is weak and can be broken), right? Could we than just use usual (not thin) manifests? -- Jauhien signature.asc Description: OpenPGP digital signature
[gentoo-dev] git security (SHA-1)
Jauhien Piatlicki: Hi, On 09/15/2014 01:37 AM, Kent Fredric wrote: On 15 September 2014 11:25, hasufell hasuf...@gentoo.org wrote: Robin said The Git commit-signing design explicitly signs the entire commit, including blob contents, to avoid this security problem. Is this correct or not? I can verify a commit by hand with only the commit object and gpg, but without any of the trees or parents. https://gist.github.com/kentfredric/8448fe55ffab7d314ecb So signing of git commits does not guarantee enough security (taking that SHA1 is weak and can be broken), right? Could we than just use usual (not thin) manifests? * there is no known SHA-1 collision afais * calculating one isn't that hard. NSA might be able to do it in reasonable time * however, the algorithms to do that will come up with random garbage, so it's a completely different thing to hide a useful vulnerability behind a SHA-1 collision
Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)
On 15 September 2014 22:10, Jauhien Piatlicki jauh...@gentoo.org wrote: So signing of git commits does not guarantee enough security (taking that SHA1 is weak and can be broken), right? Could we than just use usual (not thin) manifests? However, the attackability of SHA1 may be entirely immaterial, because methods to exploit that require compromising other security strategies. If somebody pushes signed commit 0x0001 with parents 0x0002 and 0x0003 with tree 0x0004 with files 0x0005 to 0x0010, those binary blobs are pushed. And there is no way I know of to have those binary blobs replaced with cuckoo blobs. Once they're replicated, Git doesn't try re-replicating the same SHA1s. So your attack vectors entail directly manipulating the git storage on gentoo's servers, or poisoning a mirror under their control, or poisoning the data *PRIOR* to it landing on gentoo servers, or being NSA and poisoning it dynamically when a user attempts to fetch that specific SHA1. None of these are impossible things, but they're much more complex than just make a dodgy commit and get somebody to pull it. This basically means you could use CRC32 as your hash algorithm and still pose a respectable problem for would-be attackers. As such, I don't presently see git commit signing as a Security model, merely a proof of authorship model. Anybody can forge commits with your 'author = ' and 'committer = ', but you're the only person who can sign the commit with your signature. That is to say, you sign that you crafted the *commit*. But you're *not* signing the creation of any of the dependencies. For instance, two commits may have the same tree, but obviously only one person forged that tree object. And two trees may have the same file ( and indeed, this is an expected element of how git works ), but you're not signing that you created all the files in that tree. You may infer that from the chain of authority from the commit itself, but it is not fact. And parent objects are also dependencies, and nobody would ever consider claiming they're a signing authority for all of that ;), that would be by proxy signing the creation of the entire repository back to the first commit ever forged! and its for that reason its probably good that git doesn't presently recursively feed all dependencies of a commit into GPG. I don't have 5 hours while every single blob in my repository is uncompressed and fed through GPG :p -- Kent *KENTNL* - https://metacpan.org/author/KENTNL
Re: [gentoo-dev] gentoo git workflow
Andreas K. Huettel: However, rebasing changes *on* master, before they are pushed, is a good thing, because that kills non-fast-forward merges. Nontrivial rebases *on* master can be problematic because you're changing history. Imagine you pull some nice commits from a user. Then at some point you will have to rebase them before you push them. If this fails and requires manual interaction, the original version of the commits is lost (including signatures) and errors are not traceable. With a merge instead any manual intervention is clearly located in the merge commit and the authors of each change are uniquely identifiable. So, I've tried to come up with a more detailed approach. Some people mentioned that we maybe shouldn't do rebases at all, because git already does trivial rebases automatically, but I'd try with them first. == commit policy == * atomic commits (one logical change) * commits may span across multiple ebuilds/directories if it's one logical change * every commit on the first parent of the history must be gpg signed by a gentoo dev * repoman must be run from all related directories (or the top-level directory) on the latest commit that is being pushed === commit message format === * all lines max 70-75 chars * first line brief explanation * second line always empty * optional detailed multiline explanation must start at the third line * for commits that affect only a single package, prepend CATEGORY/PN: to the first line * for commits that affect only the profile directory, prepend profiles: to the first line * for commits that affect only the eclass directory, prepend ECLASSNAME.eclass: to the first line * for commits that affect only licenses directory, prepend licenses: to the first line * for commits that affect only metadata directory, prepend metadata: to the first line == branching model == * the primary production-ready branch is master (users will pull from here), there are no non-fast-forward pushes allowed * there may be developer-specific, task-specific, project-specific branches etc (there are currently no specific rules about them) === when/how to rebase === * in case of a non-fast-forward push conflict to remote master, try 'git pull --rebase=preserve' first, if that yields complicated conflicts, abort the rebase and continue with a regular merge * use --preserve to preserve merges during a rebase (if appropriate, e.g. for user branches) * don't use --preserve if you do an interactive rebase (see BUGS in git-rebase manpage) * commits that are not on the remote master branch yet may be rewritten/squashed/splitted etc via interactive rebase, however the rebase must never span beyond those commits * never rebase on already pushed commits * there are no particular rules for rebasing on non-master remote branches, but be aware that others might base their work on them * there are no particular rules for rebasing non-remote branches, as long as they don't clutter the history when merged back into master * don't do complicated rebases to avoid a merge commit at all cost (it may even cause losing information, e.g. user signatures) === when/how to merge === * if a rebase fails or is too complicated, do a regular merge instead * if the information is useful (e.g. pulled from a foreign remote user branch or merged a non-trivial eclass conversion back into master) force a merge commit (non-fast-forward merge via '--no-ff') * to avoid a merge commit (e.g. information is not useful), you may try to force a fast-forward merge by first rebasing the feature branch (or similar) against master and then merging it into master * extend merge commit messages with useful information, e.g. how conflicts were solved * keep in mind that all commits of the first parent of the history must be gpg signed by a gentoo dev, so you may want to force merge commits especially for user branches == remote model == We have a main developer repo where developers work commit (every developer has direct push access). For every push into developer repo, automated magic thingie merges stuff into user sync repo and updates the metadata cache there. User sync repo is for power users than want to fetch via git. It's quite fast and efficient for frequent updates, and also saves space by being free of ChangeLogs. On top of user sync repo rsync is propagated. The rsync tree is populated with all old ChangeLogs copied from CVS (stored in 30M git repo), new ChangeLogs are generated from git logs and Manifests are expanded.
Re: [gentoo-dev] git security (SHA-1)
On Mon, Sep 15, 2014 at 12:35 PM, hasufell hasuf...@gentoo.org wrote: Jauhien Piatlicki: Hi, On 09/15/2014 01:37 AM, Kent Fredric wrote: On 15 September 2014 11:25, hasufell hasuf...@gentoo.org wrote: Robin said The Git commit-signing design explicitly signs the entire commit, including blob contents, to avoid this security problem. Is this correct or not? I can verify a commit by hand with only the commit object and gpg, but without any of the trees or parents. https://gist.github.com/kentfredric/8448fe55ffab7d314ecb So signing of git commits does not guarantee enough security (taking that SHA1 is weak and can be broken), right? Could we than just use usual (not thin) manifests? * there is no known SHA-1 collision afais * calculating one isn't that hard. NSA might be able to do it in reasonable time * however, the algorithms to do that will come up with random garbage, so it's a completely different thing to hide a useful vulnerability behind a SHA-1 collision sha-1 has is not directly a security feature, rather merely consistency check (albeit it has some security implications). hash is generated by using hash of commit parent and is checked every time you pull. You can actually take master commit and hash and then recursively check whole tree (albeit only basic check from current head parent is done when you pull, assuming that local tree is intact). Corrupting a single commit should be easy enough, but since the system is decentralised, people who you pull from should catch that attempt automagically. At least that's my understanding of how git (mercurial) works. Linus himself sums it nicely here: https://www.youtube.com/watch?feature=player_detailpagev=4XpnKHJAok8#t=3375 T.P.
Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)
On Mon, Sep 15, 2014 at 11:26:47PM +1200, Kent Fredric wrote: None of these are impossible things, but they're much more complex than just make a dodgy commit and get somebody to pull it. Much more simple would be to make a dodgy commit by one of the devs. Why use users for that, if the bad guy(/-s) could be inside? Isn't that the best way to poison stuff? Just look at the let's move to git threads, there're bikeshedding and it's still cvs+rsync (maybe, just maybe, this isn't a coincidence) Piotr Szymaniak. -- Mezczyzna odlozyl gazete z powrotem na stojak i postapil krok w przod, robiac mine, ktora nadala mu wyglad swinskiego pecherza rozpietego na drucianym wieszaku do garniturow. -- Graham Masterton, The Burning signature.asc Description: Digital signature
Re: [gentoo-dev] git security (SHA-1)
hasufell: * there is no known SHA-1 collision afais * calculating one isn't that hard. NSA might be able to do it in reasonable time * however, the algorithms to do that will come up with random garbage, so it's a completely different thing to hide a useful vulnerability behind a SHA-1 collision That said... an attacker who has that much resources to calculate a _random_ hash collision in reasonable time would certainly have a lot of easier attack vectors than forging a _non-random_ hash collision that contains actual working code (which, afaiu doesn't effectively work with the current attack algorithms on SHA-1). He could simply break into one of the ~200 developer computers. There's a pretty high chance at least one of them is running windows or known vulnerable versions of the kernel or other random packages. No need to waste millions of dollars on SHA-1.
Re: [gentoo-dev] gentoo git workflow
On Mon, Sep 15, 2014 at 7:37 AM, hasufell hasuf...@gentoo.org wrote: * repoman must be run from all related directories (or the top-level directory) on the latest commit that is being pushed This should be clarified. Does repoman need to be run on the exact commit that is being pushed, or perhaps on a parent commit prior to rebasing/merging into the master branch? (I use parent liberally here, since that commit wouldn't be an actual parent if it were rebased.) The reason I ask is that repoman takes a while to run (especially on the full tree), and if we don't want non-fast-forward pushes then you'll have to keep repeating the pull ; merge ; repoman ; push cycle until you finally complete it in less time than it takes somebody else to do a push. Today people do not run on the tree as it exists the instant before commit because cvs makes no guarantees about tree consistency at all (basically cvs is like git configured to accept non-fast-forward commits). == branching model == * the primary production-ready branch is master (users will pull from here), there are no non-fast-forward pushes allowed * there may be developer-specific, task-specific, project-specific branches etc (there are currently no specific rules about them) I suggest we at least toss out some kind of naming convention to prevent chaos. How about dev/name as the namespace for devs acting as individuals (devs can do whatever they want below this), and project/name as the namespace for projects (which can also do whatever they want below this). If we missed anything devs should discuss on-list before just creating random branch names. I don't really want to contrain what people do here - just try to organize it at least a tiny bit. -- Rich
Re: [gentoo-dev] gentoo git workflow
Rich Freeman: On Mon, Sep 15, 2014 at 7:37 AM, hasufell hasuf...@gentoo.org wrote: * repoman must be run from all related directories (or the top-level directory) on the latest commit that is being pushed This should be clarified. Does repoman need to be run on the exact commit that is being pushed, or perhaps on a parent commit prior to rebasing/merging into the master branch? (I use parent liberally here, since that commit wouldn't be an actual parent if it were rebased.) Yes, you have to rerun repoman after a rebase or merge. On the tip of the local master branch (as in: right before you try to push). Sure, this may lead to problems if repoman takes long... but that's on purpose. If your changes are that big, then they should be communicated and coordinated properly without people randomly pushing changes in between that may break yours. That's no different from what we are doing right now, except that we have now enforced consistency instead of maybe repoman is correct, maybe not. == branching model == * the primary production-ready branch is master (users will pull from here), there are no non-fast-forward pushes allowed * there may be developer-specific, task-specific, project-specific branches etc (there are currently no specific rules about them) I suggest we at least toss out some kind of naming convention to prevent chaos. How about dev/name as the namespace for devs acting as individuals (devs can do whatever they want below this), and project/name as the namespace for projects (which can also do whatever they want below this). If we missed anything devs should discuss on-list before just creating random branch names. I don't really want to contrain what people do here - just try to organize it at least a tiny bit. Makes sense. If there are no major disagreements, I'll open a wiki page for gentoo git workflow in a few days... as a draft ofc that is open to changes.
Re: [gentoo-dev] gentoo git workflow
On Mon, Sep 15, 2014 at 9:13 AM, hasufell hasuf...@gentoo.org wrote: Yes, you have to rerun repoman after a rebase or merge. On the tip of the local master branch (as in: right before you try to push). Sure, this may lead to problems if repoman takes long... but that's on purpose. If your changes are that big, then they should be communicated and coordinated properly without people randomly pushing changes in between that may break yours. What do you mean by proper coordination? Are you suggesting that we have scheduled downtime anytime we do a kde stablereq or something like that, once per arch? Simply announcing the change isn't going to make repoman run any faster. If you try to stablize something like kde and the packages are in more than one category, then you will need something like 10-20min of tree downtime to do a commit if you have to do a repoman scan in the middle on spinning disks. If somebody so much as does a whitespace fix on a completely unrelated package you won't get a fast-forward commit. I'm all for consistency, but repoman against the entire tree is not fast, and if you ban non-fast-forward commits then effectively all git commits collide. Combining the two makes the collision window unsuitable for commits even in the middle of the night. So, unless we want to schedule all multi-category commits and build some kind of mechanism to even enforce this, I think we'll have to live with trusting devs to just run repoman before their final merge/rebase, and then they can use their judgment about whether it will cause problems. I'm not saying that many big changes shouldn't be coordinated, but that shouldn't apply to every kde stablereq/etc. I think that this will still be a big improvement on what we have today. -- Rich
Re: [gentoo-dev] gentoo git workflow
Rich Freeman: On Mon, Sep 15, 2014 at 9:13 AM, hasufell hasuf...@gentoo.org wrote: Yes, you have to rerun repoman after a rebase or merge. On the tip of the local master branch (as in: right before you try to push). Sure, this may lead to problems if repoman takes long... but that's on purpose. If your changes are that big, then they should be communicated and coordinated properly without people randomly pushing changes in between that may break yours. What do you mean by proper coordination? Are you suggesting that we have scheduled downtime anytime we do a kde stablereq or something like that, once per arch? I don't know how kde stablereqs work. Simply announcing the change isn't going to make repoman run any faster. If you try to stablize something like kde and the packages are in more than one category, then you will need something like 10-20min of tree downtime to do a commit if you have to do a repoman scan in the middle on spinning disks. If somebody so much as does a whitespace fix on a completely unrelated package you won't get a fast-forward commit. Whether the package is related or not has to be verified by someone. I doubt that a lot of people will review the changeset after a rebase or merge and I don't have a lot of faith in people doing things right including myself. So, unless we want to schedule all multi-category commits and build some kind of mechanism to even enforce this Why all? If the multi-category commits are so big that you actually have to run repoman on top-level (you could as well run it on all related ebuild directories or on category-level), then yes... those monster pushes should definitely be scheduled. I think top-level repoman checks will be the exception, even for multi-category commits.
Re: [gentoo-dev] gentoo git workflow
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 15/09/14 09:13 AM, hasufell wrote: Rich Freeman: On Mon, Sep 15, 2014 at 7:37 AM, hasufell hasuf...@gentoo.org wrote: * repoman must be run from all related directories (or the top-level directory) on the latest commit that is being pushed This should be clarified. Does repoman need to be run on the exact commit that is being pushed, or perhaps on a parent commit prior to rebasing/merging into the master branch? (I use parent liberally here, since that commit wouldn't be an actual parent if it were rebased.) Yes, you have to rerun repoman after a rebase or merge. On the tip of the local master branch (as in: right before you try to push). Sure, this may lead to problems if repoman takes long... but that's on purpose. If your changes are that big, then they should be communicated and coordinated properly without people randomly pushing changes in between that may break yours. That's no different from what we are doing right now, except that we have now enforced consistency instead of maybe repoman is correct, maybe not. Thread TL;DR -- apologies if this has already been brought up and answered. I'm not that worried about the big (multi-package) commits, as it does make sense we're going to have difficulty and lots of potential conflicts there, but aren't we going to run into this issue just with multiple people committing separate single-package commits at the same time?? -BEGIN PGP SIGNATURE- Version: GnuPG v2 iF4EAREIAAYFAlQW9wIACgkQ2ugaI38ACPBunQEAh5F1moJZnHhHEQDs8kiPuRBk wwO2y9z6KEqKUPTzBAEA/A9Cv5Y444xC9tjmbUJ27qcRnziJgQnzu2K3YGKSCyL6 =Jc80 -END PGP SIGNATURE-
[gentoo-dev] RFC: kde5 and kde5-functions eclass
Hi,
Please find attached two new KDE eclasses for review, required to
support KDE Frameworks 5 and its consumers.
I will commit in a week or so in the absence of major issues, with the
masked packages to follow shortly after.
Best regards,
Michael
# Copyright 1999-2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
# @ECLASS: kde5.eclass
# @MAINTAINER:
# k...@gentoo.org
# @BLURB: Support eclass for KDE 5-related packages.
# @DESCRIPTION:
# The kde5.eclass provides support for building KDE 5-related packages.
if [[ -z ${_KDE5_ECLASS} ]]; then
_KDE5_ECLASS=1
CMAKE_MIN_VERSION=2.8.12
# @ECLASS-VARIABLE: VIRTUALX_REQUIRED
# @DESCRIPTION:
# For proper description see virtualx.eclass manpage.
# Here we redefine default value to be manual, if your package needs virtualx
# for tests you should proceed with setting VIRTUALX_REQUIRED=test.
: ${VIRTUALX_REQUIRED:=manual}
inherit kde5-functions toolchain-funcs fdo-mime flag-o-matic gnome2-utils
versionator virtualx eutils cmake-utils
if [[ ${KDE_BUILD_TYPE} = live ]]; then
case ${KDE_SCM} in
svn) inherit subversion ;;
git) inherit git-r3 ;;
esac
fi
EXPORT_FUNCTIONS pkg_setup src_unpack src_prepare src_configure src_compile
src_test src_install pkg_preinst pkg_postinst pkg_postrm
# @ECLASS-VARIABLE: QT_MINIMAL
# @DESCRIPTION:
# Minimal Qt version to require for the package.
: ${QT_MINIMAL:=5.3.0}
# @ECLASS-VARIABLE: KDE_AUTODEPS
# @DESCRIPTION:
# If set to false, do nothing.
# For any other value, add a dependency on dev-libs/extra-cmake-modules and
dev-qt/qtcore:5.
: ${KDE_AUTODEPS:=true}
# @ECLASS-VARIABLE: KDE_DEBUG
# @DESCRIPTION:
# If set to false, unconditionally build with -DNDEBUG.
# Otherwise, add debug to IUSE to control building with that flag.
: ${KDE_DEBUG:=true}
# @ECLASS-VARIABLE: KDE_DOXYGEN
# @DESCRIPTION:
# If set to false, do nothing.
# Otherwise, add doc to IUSE, add appropriate dependencies, and generate and
# install API documentation.
if [[ ${CATEGORY} = kde-frameworks ]]; then
: ${KDE_DOXYGEN:=true}
else
: ${KDE_DOXYGEN:=false}
fi
# @ECLASS-VARIABLE: KDE_EXAMPLES
# @DESCRIPTION:
# If set to false, unconditionally ignore a top-level examples subdirectory.
# Otherwise, add examples to IUSE to toggle adding that subdirectory.
: ${KDE_EXAMPLES:=false}
# @ECLASS-VARIABLE: KDE_HANDBOOK
# @DESCRIPTION:
# If set to false, do nothing.
# Otherwise, add +handbook to IUSE, add the appropriate dependency, and
# generate and install KDE handbook.
: ${KDE_HANDBOOK:=false}
# @ECLASS-VARIABLE: KDE_NLS
# @DESCRIPTION:
# If set to false, do nothing.
# Otherwise, add nls to IUSE, generate and install translations based on
# the LINGUAS environment variable.
if [[ ${CATEGORY} = kde-frameworks ]]; then
: ${KDE_NLS:=true}
else
: ${KDE_NLS:=false}
fi
# @ECLASS-VARIABLE: KDE_TEST
# @DESCRIPTION:
# If set to false, do nothing.
# For any other value, add test to IUSE and add a dependency on dev-qt/qttest:5.
if [[ ${CATEGORY} = kde-frameworks ]]; then
: ${KDE_TEST:=true}
else
: ${KDE_TEST:=false}
fi
if [[ ${KDEBASE} = kdevelop ]]; then
HOMEPAGE=http://www.kdevelop.org/;
else
HOMEPAGE=http://www.kde.org/;
fi
LICENSE=GPL-2
if [[ ${CATEGORY} = kde-frameworks ]]; then
SLOT=5/$(get_version_component_range 1-2)
else
SLOT=5
fi
case ${KDE_AUTODEPS} in
false) ;;
*)
if [[ ${KDE_BUILD_TYPE} = live ]]; then
ecm_version=
elif [[ ${CATEGORY} = kde-frameworks ]]; then
ecm_version=1.$(get_version_component_range 2).0
else
ecm_version=1.2.0
fi
DEPEND+= =dev-libs/extra-cmake-modules-${ecm_version}
RDEPEND+= =kde-frameworks/kf-env-2
COMMONDEPEND+= =dev-qt/qtcore-${QT_MINIMAL}:5
unset ecm_version
;;
esac
case ${KDE_DOXYGEN} in
false) ;;
*)
IUSE+= doc
DEPEND+= doc? (
$(add_frameworks_dep kapidox)
app-doc/doxygen
)
;;
esac
case ${KDE_DEBUG} in
false) ;;
*)
IUSE+= debug
;;
esac
case ${KDE_EXAMPLES} in
false) ;;
*)
IUSE+= examples
;;
esac
case ${KDE_HANDBOOK} in
false) ;;
*)
IUSE+= +handbook
DEPEND+= handbook? ( $(add_frameworks_dep kdoctools) )
;;
esac
case ${KDE_NLS} in
false) ;;
*)
IUSE+= nls
;;
esac
case ${KDE_TEST} in
false) ;;
*)
IUSE+= test
DEPEND+= test? ( dev-qt/qttest:5 )
;;
esac
DEPEND+= ${COMMONDEPEND}
Re: [gentoo-dev] gentoo git workflow
On Mon, Sep 15, 2014 at 10:26 AM, Ian Stakenvicius a...@gentoo.org wrote: I'm not that worried about the big (multi-package) commits, as it does make sense we're going to have difficulty and lots of potential conflicts there, but aren't we going to run into this issue just with multiple people committing separate single-package commits at the same time?? So, if you require fast-forward commits (which I think is sensible), then all commits collide. The other factor here is the time to run repoman. Running a repoman scan on a single package only takes tens of seconds, which means we're ok as long as we don't do commits every few seconds. But yes, you're right that commits of unrelated packages can potentially collide. The issue with something like kde is that the stablereqs can involve 100+ packages in several categories. Either you run repoman scan against all of them individually, or you run it at the tree level, and either way it is going to take a while to run (and if it is disk-bound, running in parallel probably won't help much). The reality is that a kde stabilization is not a huge risk. The packages are fairly contained, and they aren't system packages so if something does go wrong recovery is generally easy. I've yet to hear of something like this going badly. I think the worst I've seen is the odd package getting missed on the first pass. In any case, I don't have any hard numbers to prove that requiring repoman scans on every commit is a bad idea. However, I think that it has a large potential to make larger-scale changes difficult. It would make a lot more sense if we had a release-oriented strategy, even if releases were hourly/daily/etc. -- Rich
Re: [gentoo-dev] gentoo git workflow
Ian Stakenvicius: I'm not that worried about the big (multi-package) commits, as it does make sense we're going to have difficulty and lots of potential conflicts there, but aren't we going to run into this issue just with multiple people committing separate single-package commits at the same time?? Yes. If that becomes a major problem, then we have to say goodbye to the central repository workflow, because we either * allow inconsistency and broken states as we do now with CVS (and rely on QA to run a repoman tinderbox and reverse-fixing broken crap) * require devs to rebase+merge and rerun repoman 10+ times in order to get a single commit pushed It is already pretty gross and broken to have ~200 people have direct push access to a single repository. However, having ~150 commits per day I don't think this commit rate is high enough to deadlock push access. git pull --rebase=preserve repoman full git push origin master or something similar. This can even be implemented a bit more sanely within repoman or another script, but I don't think we need that yet. It's also possible that projects start accumulating changes in branches or even their own repositories to minimize the amount of pushes to master. The merge-rate will be something that has to be dynamically figured out and will improve tree quality. We shouldn't try to map our CVS workflow 1:1 to git, so there have to be changes in workflow. The optimum would be to have a real review workflow enforced with only a handful of people having direct push access, but I doubt this will work in our current community and mindset. However, such a review workflow can dynamically evolve, because it's git after all! If that happens, we can just make it a policy at some point or even implement it on infra-level.
Re: [gentoo-dev] gentoo git workflow
Rich Freeman: It would make a lot more sense if we had a release-oriented strategy, even if releases were hourly/daily/etc. If we are going that way, then we should think over the whole branching model. I have a few things in mind, but I think we are already fine-tuning stuff here that can still be fine-tuned later. The workflow will have to be adjusted after git deployment anyway. We cannot foresee everything (e.g. maybe this will be a non-issue if projects accumulate changes and push them in intervals, as pointed out in my previous post). If it doesn't work out with rerunning repoman, then we just tell people it's fine without it and think about alternative approaches, like creating additional branches, doing releases or whatnot.
Re: [gentoo-dev] gentoo git workflow
hasufell hasuf...@gentoo.org wrote: * allow inconsistency and broken states as we do now with CVS (and rely on QA to run a repoman tinderbox and reverse-fixing broken crap) ... Rich Freeman: It would make a lot more sense if we had a release-oriented strategy, even if releases were hourly/daily/etc. If we are going that way, then we should think over the whole branching model. I have a few things in mind, but I think we are already fine-tuning stuff here that can still be fine-tuned later. Apologies for replying to two different emails in one here, but these are related for the purpose of this reply. I'm all for proposals to improve the way Gentoo works. However, part of the reason that the git migration keeps on never happening is that there is this general desire out there to tie it to some kind of complete transformation in how we operate. Today, we don't have the same kind of tree-consistency you're advocating. So, moving to git without achieving this kind of consistency is not a regression. Today, we don't have some kind of completely airtight everything-is-gpg-signed secure code flow. So, moving to git without achieving that is also not a regression. If we want to have a discussion around whether Gentoo would be better off if we were more release-based (even if those releases were frequent/automated/etc), or about how to improve the security of our code base, I think those would be very healthy discussions to have. However, I don't think we should tie them to the git migration. Simply moving to git while keeping just about everything else the same will be a fairly disruptive change, and as we've already seen in this thread there are some who just prefer cvs (though I think they're in the minority by far). If we try to make several other big changes at the same time I just think that it will never happen. I suggest we just get git working in a fashion that is good enough. I think that will already bring an increased level of consistency compared to our current cvs workflow, where people run repoman from cvs checkouts where you can't even ensure that any two devs have quite the same trees at the same time. There is nothing that keeps us from moving from that model to one that is more revolutionary at another time. However, trying to do it all at once probably means that none of us get anything we're looking for, because we're not going to do it with cvs and we'll never get off of cvs unless simply doing that by any means necessary is the imperative. -- Rich
Re: [gentoo-dev] gentoo git workflow
Rich Freeman: I suggest we just get git working in a fashion that is good enough. Sure, that's what I've been saying. Otherwise I'd propose to remove access for everyone and only grant project leads/reviewers direct push access. But as said... we are not ready for something like that. However, gpg signature verification is trivial to implement (and is already) and starting with stricter policies is worth a try. Nothing of that is really a big deal. But... starting release-oriented strategies IS a big deal.
Re: [gentoo-dev] RFC: kde5 and kde5-functions eclass
kde5-functions.eclass
inherit versionator
versionator doesn't export any phase functions so it can stay inside the
_KDE5_FUNCTIONS_ECLASS conditional block.
case ${EAPI:-0} in
I believe :-0 is unnecessary here, *) will match anyway, but it doesn't
hurt either.
*) die EAPI=${EAPI} is not supported ;;
Here ${EAPI:-0} is needed instead.
if [[ ${CATEGORY} = kde-base ]]; then
debug-print ${ECLASS}: KDEBASE ebuild recognized
KDEBASE=kde-base
elif [[ ${CATEGORY} = kde-frameworks ]]; then
debug-print ${ECLASS}: KDEFRAMEWORKS ebuild recognized
KDEBASE=kde-frameworks
elif [[ ${KMNAME-${PN}} = kdevelop ]]; then
debug-print ${ECLASS}: KDEVELOP ebuild recognized
KDEBASE=kdevelop
fi
Can you move the print after the if-fi block to avoid duplication?
debug-print ${ECLASS}: ${KDEBASE} ebuild recognized
case ${KDE_SCM} in
svn|git) ;;
I thought : was necessary for syntax reasons but apparently it's not. You
may want to add it anyway for consistency with the previous case statement.
if [[ -a CMakeLists.txt ]]; then
Unnecessary quoting. Also, -e is more common than -a
sed -e
/add_subdirectory[[:space:]]*([[:space:]]*${1}[[:space:]]*)/s/^/#DONOTCOMPILE
/ \
What if ${1} contains slashes?
[[ -z ${1} ]] die Missing parameter
Sanity checking of arguments should be done at the beginning of the
function body. Also, is $2 allowed to be empty?
# @FUNCTION: add_frameworks_dep
Add @USAGE eclass doc.
local ver=${1:-${PV}}
local major=$(get_major_version ${ver})
local minor=$(get_version_component_range 2 ${ver})
local micro=$(get_version_component_range 3 ${ver})
if [[ ${ver} == ]]; then
No wildcard match here? (*)
Thanks,
Davide
Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 14/09/14 07:21 PM, Patrick Lauer wrote: On Sunday 14 September 2014 15:42:15 hasufell wrote: Patrick Lauer: Are we going to disallow merge commits and ask devs to rebase local changes in order to keep the history clean? Is that going to be sane with our commit frequency? You have to merge or rebase anyway in case of a push conflict, so the only difference is the method and the effect on the history. Currently... CVS allows you to run repoman on an outdated tree and push broken ebuilds with repoman being happy. Git will not allow this. iow, git doesn't allow people to work on more than one item at a time? That'd mean I need half a dozen checkouts just to emulate cvs, which somehow doesn't make much sense to me ... I think you'd just need to have a (large) handful of different branches, and squash+cherry-pick into master when you're ready to push. -BEGIN PGP SIGNATURE- Version: GnuPG v2 iF4EAREIAAYFAlQXH08ACgkQ2ugaI38ACPAZ1gD/WfiMZnu3qesaILhPYEKYy2BP MUS2zWJVqYJ8lKp16nUA/1ng1mMxX6pNKZVYIaT/BFuERKz3g0BcLck+XILs3Hth =Ul7F -END PGP SIGNATURE-
Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 14/09/14 08:57 PM, Rich Freeman wrote: On Sun, Sep 14, 2014 at 7:21 PM, Patrick Lauer patr...@gentoo.org wrote: iow, git doesn't allow people to work on more than one item at a time? That'd mean I need half a dozen checkouts just to emulate cvs, which somehow doesn't make much sense to me ... Well, you can work on as many things as you like in git, but it doesn't keep track of what changes have to do with what things if you don't commit in-between. So, you'll have a big list of changes in your index, and you'll have to pick-and-choose what you commit at any one time. If you really want to work on many things at once the better way to do it is to do a temporary branch per-thing, and when you switch between things you switch between branches, and then move into master things as they are done. I assume you mean working on things that will take a while to complete. If you just want to do 15 standalone commits before you push you can do those sequentially easily enough. A branch would be more appropriate for some kind of mini-project. You can work on branches without pushing those to the master repo. Or, if appropriate a project team might choose to push their branch to master, or to some other repo (like an overlay). This would allow collaborative work on a large commit, with a quick final merge into the main tree. That is the beauty of git - branches are really cheap. So are repositories - if somebody wants to do all their work in github and then push to the main tree, they can do that. Actually i see what Patrick's getting at -- I have similar issues when working with mozilla stuff. if you're using local (temporary) branches, the whole tree is in the state of that current checkout, right? IE, while I have my firefox-update branch active and working for an 'ebuild ... install', I can't be doing work in my 'freewrl-update' branch unless I make multiple separate repo trees, one for each independently-separate workflow i want to do concurrently. Ideal here would be the ability to have separate active checkouts of a repo on a per-shell basis, ie each shell invocation would be able to work concurrently and distinctly on distinct branches; anyone done that before? Does git do it already? -BEGIN PGP SIGNATURE- Version: GnuPG v2 iF4EAREIAAYFAlQXJAAACgkQ2ugaI38ACPBREAD/YnsyY+fAK1TEXgzNYHBCq138 Q5Bj+J6pNGX8aBDjjHoA/iyy5CWxhyAYE3buSOXkEvFfhm/716DsQIptpX7JpS0m =YrIG -END PGP SIGNATURE-
Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 14/09/14 09:06 PM, Peter Stuge wrote: Rich Freeman wrote: If you just want to do 15 standalone commits before you push you can do those sequentially easily enough. A branch would be more appropriate for some kind of mini-project. .. That is the beauty of git - branches are really cheap. So are repositories And commits. Not only are branches cheap, they are also very easy to create, and maybe most importantly they can be created at any time, even after the commits. It's quick and painless to create a bunch of commits which aren't really closely related in sequence, and only later clean the whole series of commits up while creating different branches for commits which should actually be grouped rather than mixed all together. Ahh, so the secret here would then be just to git add files on a related per-package basis, leaving the other files out of the commit. that makes sense. There would still be the issue of untracked files in the repo and the ability to switch back to the 'master' branch to cherry-pick a commit for pushing, though... I guess we'd just have to deal with the delay there and try and push all of the changes at once? -BEGIN PGP SIGNATURE- Version: GnuPG v2 iF4EAREIAAYFAlQXJQkACgkQ2ugaI38ACPBFBQD/Z1SYvajcf/WORxknJGu1VfI0 f8CFhMTdE34Bk0Zd+GoA/iJtwsYBUQQHXhRjs7AzQDxaIEuFRgzyUgee4BICKaiq =8fbP -END PGP SIGNATURE-
Re: [gentoo-dev] git basics
Ian Stakenvicius: On 14/09/14 09:06 PM, Peter Stuge wrote: Rich Freeman wrote: If you just want to do 15 standalone commits before you push you can do those sequentially easily enough. A branch would be more appropriate for some kind of mini-project. .. That is the beauty of git - branches are really cheap. So are repositories And commits. Not only are branches cheap, they are also very easy to create, and maybe most importantly they can be created at any time, even after the commits. It's quick and painless to create a bunch of commits which aren't really closely related in sequence, and only later clean the whole series of commits up while creating different branches for commits which should actually be grouped rather than mixed all together. Ahh, so the secret here would then be just to git add files on a related per-package basis, leaving the other files out of the commit. that makes sense. There would still be the issue of untracked files in the repo and the ability to switch back to the 'master' branch to cherry-pick a commit for pushing, though... I guess we'd just have to deal with the delay there and try and push all of the changes at once? First, I don't think this is related to mgornys migration plans anymore. Second: yes, in contrary to CVS it will _never_ happen on git that you accidentally commit random files because of a dirty tree (and that happened to me as well on CVS... I didn't even know until I ran repoman on a whole category by accident! oops). There are numerous of ways to work on multiple things at once in git: 1) Just use different branches and sync them if necessary. 2) Don't stage everything you have modified... I guess a lot of people don't really know that it's possible? As pointed out before, you can even say which hunks/lines of a single file you want to have staged. Furthermore, you don't even need to stash anything, even if you switch branches, as long as the (un)staged changes don't collide with the branch you are switching to (and if that is the case... then you have bigger problems). 3) Although this is anti-git, you could still just copy paste the repository directory. Every git repository is consistent in itself and fully functional. You don't directly depend on any server. And you can even pull from local git repositories. If you find this silly, do 2) or 1).
Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)
On Mon, Sep 15, 2014 at 1:42 PM, Ian Stakenvicius a...@gentoo.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 14/09/14 09:06 PM, Peter Stuge wrote: Rich Freeman wrote: If you just want to do 15 standalone commits before you push you can do those sequentially easily enough. A branch would be more appropriate for some kind of mini-project. .. That is the beauty of git - branches are really cheap. So are repositories And commits. Not only are branches cheap, they are also very easy to create, and maybe most importantly they can be created at any time, even after the commits. It's quick and painless to create a bunch of commits which aren't really closely related in sequence, and only later clean the whole series of commits up while creating different branches for commits which should actually be grouped rather than mixed all together. Ahh, so the secret here would then be just to git add files on a related per-package basis, leaving the other files out of the commit. that makes sense. There would still be the issue of untracked files in the repo and the ability to switch back to the 'master' branch to cherry-pick a commit for pushing, though... I guess we'd just have to deal with the delay there and try and push all of the changes at once? If you really like to keep a lot of things going at once, I'd strongly recommend doing it in another branch (or more than one), so that you can cherry-pick/merge/etc stuff into master and not have a mess of an index while doing it. You need not publish that branch, and if you do publish it you need not do it on Gentoo's infra (a git repository can be synced with multiple other repositories, and the various heads don't need to stay in sync). In general you want each commit to represent a single change. That might be a revbump in a single package, or it might be a package move that involves touching 300 packages in a single commit. The idea though is that every commit should stand on its own, so that they can be reverted on their own, etc. That's just good practice, and should be what we're trying to do with cvs with the huge limitation that multi-file changes in cvs aren't atomic. There are a lot of guides/tools/etc out there for using git. A popular one is the git-flow workflow. I'm not suggesting that it is really necessary for one person, but anybody not familiar with git should probably read up on it just so that you have some sense of how it can be used. -- Rich
Re: [gentoo-dev] git basics
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 15/09/14 02:35 PM, hasufell wrote:
Ian Stakenvicius:
On 14/09/14 09:06 PM, Peter Stuge wrote:
Rich Freeman wrote:
If you just want to do 15 standalone commits before you push
you can do those sequentially easily enough. A branch would
be more appropriate for some kind of mini-project.
..
That is the beauty of git - branches are really cheap. So
are repositories
And commits.
Not only are branches cheap, they are also very easy to
create, and maybe most importantly they can be created at any
time, even after the commits.
It's quick and painless to create a bunch of commits which
aren't really closely related in sequence, and only later clean
the whole series of commits up while creating different
branches for commits which should actually be grouped rather
than mixed all together.
Ahh, so the secret here would then be just to git add files on a
related per-package basis, leaving the other files out of the
commit. that makes sense. There would still be the issue of
untracked files in the repo and the ability to switch back to the
'master' branch to cherry-pick a commit for pushing, though... I
guess we'd just have to deal with the delay there and try and
push all of the changes at once?
First, I don't think this is related to mgornys migration plans
anymore.
Completely agree; it may not even be on-topic for the git workflow
thread..
Second: yes, in contrary to CVS it will _never_ happen on git that
you accidentally commit random files because of a dirty tree (and
that happened to me as well on CVS... I didn't even know until I
ran repoman on a whole category by accident! oops).
There are numerous of ways to work on multiple things at once in
git: 1) Just use different branches and sync them if necessary.
2) Don't stage everything you have modified... I guess a lot of
people don't really know that it's possible? As pointed out before,
you can even say which hunks/lines of a single file you want to
have staged.
Furthermore, you don't even need to stash anything, even if you
switch branches, as long as the (un)staged changes don't collide
with the branch you are switching to (and if that is the case...
then you have bigger problems).
My concern/question is not so much how to deal with different things
concurrently, as it is how to deal with having to do distinct git
operations concurrently.
Repeating my example, say i'm working on a new release of firefox, it
takes ~40 mins to compile and there's some stuff it needs to do with
files in ${FILESDIR} during src_install. So i'm 'ebuild ...
install'ing that. In the meantime, there's a high-priority fix that
came up that I have to do in say, eudev. Unless i'm doing my firefox
work in the master branch locally (and probably am running these tests
on modified-but-not-committed files), i can't just switch branches to
do/test/commit/push while ebuild does its thing in the background. Or
can I? I'd have to resort to (3), right?
3) Although this is anti-git, you could still just copy paste the
repository directory. Every git repository is consistent in itself
and fully functional. You don't directly depend on any server. And
you can even pull from local git repositories. If you find this
silly, do 2) or 1).
If it were possible to just start up a new shell, cd /usr/portage.git;
git checkout master ; [..do stuff..] while the current workspace in
my old shell remains consistent, all would be fine. But so far as i
(and everyone currently active in #-dev) are aware there's no built-in
way to do that right now; the 'git' way would be to clone local repos,
yes?
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
iF4EAREIAAYFAlQXPgcACgkQ2ugaI38ACPCJFQD+JfDz8KfuRb5+Ybij11fAOBQv
75jah41LD1o2F7AvH4oA/3UAqtO90XS5nin+3o1qU8gfDlL6FPn5E9YGDEPtnQ6f
=Eo0Y
-END PGP SIGNATURE-
Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)
On Mon, Sep 15, 2014 at 09:53:43AM +0200, Fabian Groffen wrote: On 14-09-2014 16:56:24 +0200, Michał Górny wrote: Rich Freeman ri...@gentoo.org napisał(a): So, I don't really have a problem with your design. I still question whether we still need to be generating changelogs - they seem incredibly redundant. But, if people really want a redundant copy of the git log, whatever... I don't want them too. However, I'm pretty sure people will bikeshed this to death if we kill them... Especially that rsync has no git log. Not that many users make real use of ChangeLogs, esp. considering how useless messages often are there... Council had some discussions on this topic: http://www.gentoo.org/proj/en/council/meeting-logs/2008-summary.txt http://www.gentoo.org/proj/en/council/meeting-logs/20111011-summary.txt Conclusion back then was that ChangeLog files need to stay. I would have no problem with the council revisiting/changing this. I tend to agree that the ChangeLogs in the portage tree will be obsoleted when we switch to git because git's logging facilities are much easier to use than those in CVS. Not to mention how much smaller the portage tree would be without ChangeLogs. William signature.asc Description: Digital signature
Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)
On 09/15/14 15:30, William Hubbs wrote: On Mon, Sep 15, 2014 at 09:53:43AM +0200, Fabian Groffen wrote: On 14-09-2014 16:56:24 +0200, Michał Górny wrote: Rich Freeman ri...@gentoo.org napisał(a): So, I don't really have a problem with your design. I still question whether we still need to be generating changelogs - they seem incredibly redundant. But, if people really want a redundant copy of the git log, whatever... I don't want them too. However, I'm pretty sure people will bikeshed this to death if we kill them... Especially that rsync has no git log. Not that many users make real use of ChangeLogs, esp. considering how useless messages often are there... Council had some discussions on this topic: http://www.gentoo.org/proj/en/council/meeting-logs/2008-summary.txt http://www.gentoo.org/proj/en/council/meeting-logs/20111011-summary.txt Conclusion back then was that ChangeLog files need to stay. I would have no problem with the council revisiting/changing this. I tend to agree that the ChangeLogs in the portage tree will be obsoleted when we switch to git because git's logging facilities are much easier to use than those in CVS. Not to mention how much smaller the portage tree would be without ChangeLogs. William If the argument is that there are no Changelogs in rsync, then let's write git hooks to generate them when the repository is mirrored to the rsync host. The only problem I see is with this is then adding ChangeLog to the manifest and gpg signing it which has to be done at the developer's side. But, I think the tree that users get from rsync should have the logs. Having *both* a ChangeLog file and git log is redundant. -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA
Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)
On Mon, Sep 15, 2014 at 3:55 PM, Anthony G. Basile bluen...@gentoo.org wrote: On 09/15/14 15:30, William Hubbs wrote: I would have no problem with the council revisiting/changing this. I tend to agree that the ChangeLogs in the portage tree will be obsoleted when we switch to git because git's logging facilities are much easier to use than those in CVS. Not to mention how much smaller the portage tree would be without ChangeLogs. William If the argument is that there are no Changelogs in rsync, then let's write git hooks to generate them when the repository is mirrored to the rsync host. The only problem I see is with this is then adding ChangeLog to the manifest and gpg signing it which has to be done at the developer's side. But, I think the tree that users get from rsync should have the logs. Having *both* a ChangeLog file and git log is redundant. I'll add this to the next Council agenda. I think this is ripe for discussion. The last discussion of this really wasn't aimed at git anyway. -- Rich
Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)
Dnia 2014-09-15, o godz. 15:55:35 Anthony G. Basile bluen...@gentoo.org napisał(a): On 09/15/14 15:30, William Hubbs wrote: On Mon, Sep 15, 2014 at 09:53:43AM +0200, Fabian Groffen wrote: On 14-09-2014 16:56:24 +0200, Michał Górny wrote: Rich Freeman ri...@gentoo.org napisał(a): So, I don't really have a problem with your design. I still question whether we still need to be generating changelogs - they seem incredibly redundant. But, if people really want a redundant copy of the git log, whatever... I don't want them too. However, I'm pretty sure people will bikeshed this to death if we kill them... Especially that rsync has no git log. Not that many users make real use of ChangeLogs, esp. considering how useless messages often are there... Council had some discussions on this topic: http://www.gentoo.org/proj/en/council/meeting-logs/2008-summary.txt http://www.gentoo.org/proj/en/council/meeting-logs/20111011-summary.txt Conclusion back then was that ChangeLog files need to stay. I would have no problem with the council revisiting/changing this. I tend to agree that the ChangeLogs in the portage tree will be obsoleted when we switch to git because git's logging facilities are much easier to use than those in CVS. Not to mention how much smaller the portage tree would be without ChangeLogs. If the argument is that there are no Changelogs in rsync, then let's write git hooks to generate them when the repository is mirrored to the rsync host. The only problem I see is with this is then adding ChangeLog to the manifest and gpg signing it which has to be done at the developer's side. But, I think the tree that users get from rsync should have the logs. Having *both* a ChangeLog file and git log is redundant. Can't we just kill rsync then? The whole ChangeLog seems to take more effort than the actual benefit it gives. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] git basics
Ian Stakenvicius:
Repeating my example, say i'm working on a new release of firefox, it
takes ~40 mins to compile and there's some stuff it needs to do with
files in ${FILESDIR} during src_install. So i'm 'ebuild ...
install'ing that. In the meantime, there's a high-priority fix that
came up that I have to do in say, eudev. Unless i'm doing my firefox
work in the master branch locally (and probably am running these tests
on modified-but-not-committed files), i can't just switch branches to
do/test/commit/push while ebuild does its thing in the background. Or
can I? I'd have to resort to (3), right?
1. git checkout -b eudev-tmp (create new branch which has the firefox
changes you just made and switch to it)
2. git reset --mixed HEAD~1 (dump the firefox patch back to the unstaged
area, the file is still the same!)
3. fiddle around with eudev, commit and push
4. git checkout -f master (switch back to the master branch and throw
away the unstaged changes in eudev-tmp, however... file is still the same)
Alternatives:
Copy over the firefox ebuild to your local overlay before doing
extensive tests and fiddling. That's what I'd do anyway instead of
working directly on the tree.
Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)
On Mon, Sep 15, 2014 at 10:18:39PM +0200, Michał Górny wrote: Dnia 2014-09-15, o godz. 15:55:35 Anthony G. Basile napisał(a): If the argument is that there are no Changelogs in rsync, then let's write git hooks to generate them when the repository is mirrored to the rsync host. The only problem I see is with this is then adding ChangeLog to the manifest and gpg signing it which has to be done at the developer's side. But, I think the tree that users get from rsync should have the logs. Having *both* a ChangeLog file and git log is redundant. Can't we just kill rsync then? The whole ChangeLog seems to take more effort than the actual benefit it gives. I'm +1 for killing rsync and having everyone use Git. With --shallow clones for folks who don't care about the history, and deep clones for those who do (and you can change your mind both ways), I think everyone gets what they want without messing around with a Git → rsync conversion layer. Of course, it would be nice if the CSV → Git migration added any ChangeLog notes to the associated commit message to avoid losing information, but I imagine it would be hard to automate that and still get readable commit messages ;). I don't see any benefit to using rsync vs. a shallow clone as the transmission protocol. Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] git basics
hasufell:
Ian Stakenvicius:
Repeating my example, say i'm working on a new release of firefox, it
takes ~40 mins to compile and there's some stuff it needs to do with
files in ${FILESDIR} during src_install. So i'm 'ebuild ...
install'ing that. In the meantime, there's a high-priority fix that
came up that I have to do in say, eudev. Unless i'm doing my firefox
work in the master branch locally (and probably am running these tests
on modified-but-not-committed files), i can't just switch branches to
do/test/commit/push while ebuild does its thing in the background. Or
can I? I'd have to resort to (3), right?
1. git checkout -b eudev-tmp (create new branch which has the firefox
changes you just made and switch to it)
2. git reset --mixed HEAD~1 (dump the firefox patch back to the unstaged
area, the file is still the same!)
3. fiddle around with eudev, commit and push
4. git checkout -f master (switch back to the master branch and throw
away the unstaged changes in eudev-tmp, however... file is still the same)
Alternatives:
Copy over the firefox ebuild to your local overlay before doing
extensive tests and fiddling. That's what I'd do anyway instead of
working directly on the tree.
That said... I find that workflow a bit weird. Why is the firefox ebuild
already committed if it hasn't even been tested?
And if it's not committed... then we don't even have a problem here,
because switching branches will just work.
Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)
On Mon, Sep 15, 2014 at 01:29:44PM -0700, W. Trevor King wrote: I don't see any benefit to using rsync vs. a shallow clone as the transmission protocol. Other than the fact that before you dropped it you'd need to push a ‘emerge sync’ that could handle either rsync or Git, stabilize that Portage, and then wait for folks to adopt it. That's going to slow down your migration a bit ;). I think an rsync-able version is a better choice for the migration, but since it's not destined to live long (in my view), I don't think it really matters what goes into it. Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)
On Mon, Sep 15, 2014 at 4:18 PM, Michał Górny mgo...@gentoo.org wrote: Can't we just kill rsync then? The whole ChangeLog seems to take more effort than the actual benefit it gives. I'm not sure ditching rsync entirely is necessary - it might be more trouble than it is worth as it is a very effective simple way to distribute the tree. However, I'm not really opposed to it either. However, I do really question whether we need changelogs in rsync. It seems like many projects are going away from these - or doing what the kernel is doing and just dumping a git log into them. I don't think we need to try to shoehorn the old changelogs into our git history - I'd just leave them in the tree for migration and then prune then post-migration. Oh, in case it is useful to know, a full historical git bundle is about 1.2GB, and a clone+checkout of the bundle uses about 2.1GB of space. A compressed cvs tarball with the full history is about 575MB in comparison, though I see it has grown by about 50MB in the last six months. Bottom line is that non-shallow checkouts will need a decent amount of space. Then again, my tmpfs /usr/portage uses 735M just by itself. -- Rich
Re: [gentoo-dev] git basics
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 15/09/14 04:31 PM, hasufell wrote:
hasufell:
Ian Stakenvicius:
Repeating my example, say i'm working on a new release of
firefox, it takes ~40 mins to compile and there's some stuff it
needs to do with files in ${FILESDIR} during src_install. So
i'm 'ebuild ... install'ing that. In the meantime, there's a
high-priority fix that came up that I have to do in say, eudev.
Unless i'm doing my firefox work in the master branch locally
(and probably am running these tests on
modified-but-not-committed files), i can't just switch branches
to do/test/commit/push while ebuild does its thing in the
background. Or can I? I'd have to resort to (3), right?
1. git checkout -b eudev-tmp (create new branch which has the
firefox changes you just made and switch to it) 2. git reset
--mixed HEAD~1 (dump the firefox patch back to the unstaged area,
the file is still the same!) 3. fiddle around with eudev, commit
and push 4. git checkout -f master (switch back to the master
branch and throw away the unstaged changes in eudev-tmp,
however... file is still the same)
It's generally considered safe to push to origin/master a commit from
a temporary local branch? how would that end up working with the
repoman-push-[fail]-pull cycle due to possible non-fast-forward
conflicts? how difficult will it be to ensure local history isn't all
messed up, later on?
Alternatives: Copy over the firefox ebuild to your local overlay
before doing extensive tests and fiddling. That's what I'd do
anyway instead of working directly on the tree.
Yes, that's always a possibility as well -- having a second copy of
the tree, pulling (local) changes from the working copy and
merge-testing from that would reduce most of these concurrency
issues.. Probably worthwhile suggesting this in the git workflow
guide, as I expect there are plenty of people that test ebuilds
straight from CVS right now.
That said... I find that workflow a bit weird. Why is the firefox
ebuild already committed if it hasn't even been tested?
That's just how i tend to work with git in what little i've been
exposed to: git checkout -b tmpbranch ; develop; commit; test
{squash,checkout master,cherry-pick,push} || develop []
It lets me roll back to earlier commits if i want to undo something,
or revisit some code that i discarded, or whatnot... Since committing
!= pushing, generally there isn't much harm, and i squash all these
misc commits into one after everything's ready, so it doesn't clutter
the history.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
iF4EAREIAAYFAlQXUkoACgkQ2ugaI38ACPAmvQD9GeqZu6dHi5BSd2lpArMvHES/
whHLDOoPdXUeb3Uh99kBAKorCvYZFmKEDHWgvbnCuk/vfua58ItwOjtQIY/WG1Az
=pNOG
-END PGP SIGNATURE-
Re: [gentoo-dev] git basics
Ian Stakenvicius:
It's generally considered safe to push to origin/master a commit from
a temporary local branch?
Why not?
Even if you have to rebase/merge, nothing will happen with your unstaged
local changes as long as no one has messed with the firefox ebuild in
the meantime... and then you have bigger problems.
as I expect there are plenty of people that test ebuilds
straight from CVS right now.
I remember people telling me to never do that, but anyway.
That said... I find that workflow a bit weird. Why is the firefox
ebuild already committed if it hasn't even been tested?
That's just how i tend to work with git in what little i've been
exposed to: git checkout -b tmpbranch ; develop; commit; test
{squash,checkout master,cherry-pick,push} || develop []
It lets me roll back to earlier commits if i want to undo something,
or revisit some code that i discarded, or whatnot... Since committing
!= pushing, generally there isn't much harm, and i squash all these
misc commits into one after everything's ready, so it doesn't clutter
the history.
Maybe you could check if you can do some of that stuff with git stashes.
You can save them with a message indicating what it is about, apply a
particular one from the stash (ofc you can have multiple ones), drop
others, create branches based on the stash etc.
Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)
Rich Freeman: I'm not sure ditching rsync entirely is necessary - it might be more trouble than it is worth as it is a very effective simple way to distribute the tree. However, I'm not really opposed to it either. The few people I personally know who use gentoo never use rsync for syncing, because I told them to never use it, unless they want random backdoors in their tree.
Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)
On 09/15/14 16:49, Rich Freeman wrote: On Mon, Sep 15, 2014 at 4:18 PM, Michał Górny mgo...@gentoo.org wrote: Can't we just kill rsync then? The whole ChangeLog seems to take more effort than the actual benefit it gives. I'm not sure ditching rsync entirely is necessary - it might be more trouble than it is worth as it is a very effective simple way to distribute the tree. However, I'm not really opposed to it either. I can live with git only but I'm not sure what would happen if we tried this? There are lots of users and scripts out there that assume rsync. That's one cold shower. However, I do really question whether we need changelogs in rsync. It seems like many projects are going away from these - or doing what the kernel is doing and just dumping a git log into them. I don't think we need to try to shoehorn the old changelogs into our git history - I'd just leave them in the tree for migration and then prune then post-migration. We could just push out the word that ChangeLogs are going away and they have to read the git repo. That might be the easiest solution. I do have users that quote my ChangeLogs though. Oh, in case it is useful to know, a full historical git bundle is about 1.2GB, and a clone+checkout of the bundle uses about 2.1GB of space. A compressed cvs tarball with the full history is about 575MB in comparison, though I see it has grown by about 50MB in the last six months. Bottom line is that non-shallow checkouts will need a decent amount of space. Then again, my tmpfs /usr/portage uses 735M just by itself. -- Rich -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA
Re: [gentoo-dev] git security (SHA-1)
On Mon, Sep 15, 2014 at 7:02 AM, hasufell hasuf...@gentoo.org wrote: hasufell: * there is no known SHA-1 collision afais * calculating one isn't that hard. NSA might be able to do it in reasonable time * however, the algorithms to do that will come up with random garbage, so it's a completely different thing to hide a useful vulnerability behind a SHA-1 collision That said... an attacker who has that much resources to calculate a _random_ hash collision in reasonable time would certainly have a lot of easier attack vectors than forging a _non-random_ hash collision that contains actual working code (which, afaiu doesn't effectively work with the current attack algorithms on SHA-1). He could simply break into one of the ~200 developer computers. There's a pretty high chance at least one of them is running windows or known vulnerable versions of the kernel or other random packages. No need to waste millions of dollars on SHA-1. Even if you wanted to burn the money to find that magical collision that actually contains working code, you've still got to somehow propagate that to other repositories, since they'll just ignore it for having the same hash as an already-existing object.
Re: [gentoo-dev] git security (SHA-1)
On Tue, Sep 16, 2014 at 5:11 AM, Gordon Pettey petteyg...@gmail.com wrote: On Mon, Sep 15, 2014 at 7:02 AM, hasufell hasuf...@gentoo.org wrote: hasufell: * there is no known SHA-1 collision afais * calculating one isn't that hard. NSA might be able to do it in reasonable time * however, the algorithms to do that will come up with random garbage, so it's a completely different thing to hide a useful vulnerability behind a SHA-1 collision That said... an attacker who has that much resources to calculate a _random_ hash collision in reasonable time would certainly have a lot of easier attack vectors than forging a _non-random_ hash collision that contains actual working code (which, afaiu doesn't effectively work with the current attack algorithms on SHA-1). He could simply break into one of the ~200 developer computers. There's a pretty high chance at least one of them is running windows or known vulnerable versions of the kernel or other random packages. No need to waste millions of dollars on SHA-1. Even if you wanted to burn the money to find that magical collision that actually contains working code, you've still got to somehow propagate that to other repositories, since they'll just ignore it for having the same hash as an already-existing object. In the fetch/pull case, if you receive the same object that you already have, git performs byte-to-byte comparison and warns loudly if the newobject does not match yours. -- Duy
Re: [gentoo-dev] git security (SHA-1)
On Tue, Sep 16, 2014 at 5:41 AM, Duy Nguyen pclo...@gmail.com wrote: Even if you wanted to burn the money to find that magical collision that actually contains working code, you've still got to somehow propagate that to other repositories, since they'll just ignore it for having the same hash as an already-existing object. In the fetch/pull case, if you receive the same object that you already have, git performs byte-to-byte comparison and warns loudly if the newobject does not match yours. Correction, in the push case as well. You have to find another way to inject this collided object (perhaps via a server that is not C Git). -- Duy
Re: [gentoo-dev] git security (SHA-1)
On Mon, Sep 15, 2014 at 6:11 PM, Gordon Pettey petteyg...@gmail.com wrote: Even if you wanted to burn the money to find that magical collision that actually contains working code, you've still got to somehow propagate that to other repositories, since they'll just ignore it for having the same hash as an already-existing object. Well, if you're willing to trust that nobody is able to tamper with repositories, then you don't need gpg signatures in the first place. I think that gpg signatures protected by an SHA1 hash provide fairly little security - a chain is as strong as its weakest link and sha1 has been considered fairly weak for years now. However, I think it does make sense to at least get gpg into the workflow in the hopes that some day git will move to a stronger hash, and since it isn't a huge hardship to do so. I wouldn't make too light of the use of SHA1 though. As you point out simply exploiting it isn't enough, but the whole reason for having signatures is to make an attack on a central repository useless. Having gpg on top of ssh keys and all that is obviously redundant, but that is the whole point of it. -- Rich
[gentoo-dev] Re: RFC: kde5 and kde5-functions eclass
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512
On 09/15/2014 12:19 PM, Davide Pesavento wrote:
kde5-functions.eclass
local ver=${1:-${PV}} local major=$(get_major_version ${ver})
local minor=$(get_version_component_range 2 ${ver}) local
micro=$(get_version_component_range 3 ${ver}) if [[ ${ver} ==
]]; then
No wildcard match here? (*)
This is intentional, and comes from the kde4-functions equivalent.
The KDE version of the 5.0 branch is 5.0, not live. Only the
master/trunk/etc. branch of the live repo gets the live designation.
(Under KDE SC 4, we used versions like 4.14.49. to reference the
4.14 branch, as upstream liked to use versions like 4.14.90 for a
beta/rc of 4.15; I haven't been involved enough with KDE5 to determine
what they're doing now.)
- --
Jonathan Callen
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
iQIcBAEBCgAGBQJUF5MnAAoJELHSF2kinlg4mm0P/1DFUilFc8IfYNIsJqqK96Ec
hWrrDwK1Hy0yxCfJE4NccX7OGJjJDRi7r0O1XnEPc0vueXlCsTCzHFkOZDJOXz0M
+B6bbJm5WJCz9GYrU4rIY/BflSARPzCh+YZoEUu5Azvu2DuZkh3kHYw136mqnrvT
woqTAMPWaPZPQXMk3r8dmdSsjjtY+eplL68TSYh4x8p6qgSW/19UZcF0HjlfvHSd
JfrJ+TKupXa7i77PMOsR0GkHztVUyFbu2vVYnkPI52W7dLwlv9VuFDNGjprRl4ve
Xa5PzLGcz9Xr9JAtWSdPaTycRfkNpmYKJYwLRoOVIhJ3FvuZdT2ZD+P3x+uetem5
QumharQhkt5UNrwkWYC/KgVEJsLb0nVAYMDYUkaDTbYv1inEN6NkACjZrqvYTuSm
8zt4P9hhe73I7AliUD0AI70muvlOrGaseDXSrD0thsibqPHpK215IUgVyCNQjQ7U
NRu73bL2cEQ9fuoxzKmfTY3oPl5B9W9J3sXiKU4SEbWzplhJRQ2zBX1ZyqNgJMiP
ZZH1XWAiaotUzsrALcHcJ1iCCFE2rloIMA8unuxFhh7ch5MkMKsWfMLXio/pudTW
A0pHbnTAWonWIWdTy5HjBe7wUxu5Wxb09S5UbgCsnI6QhhorBNwvWfy/RGqdUThl
EUYzDpG+4ImziiPSm9Yu
=73JG
-END PGP SIGNATURE-
Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)
On 15-09-2014 15:58:00 -0400, Rich Freeman wrote: If the argument is that there are no Changelogs in rsync, then let's write git hooks to generate them when the repository is mirrored to the rsync host. The only problem I see is with this is then adding ChangeLog to the manifest and gpg signing it which has to be done at the developer's side. But, I think the tree that users get from rsync should have the logs. Having *both* a ChangeLog file and git log is redundant. I'll add this to the next Council agenda. I think this is ripe for discussion. The last discussion of this really wasn't aimed at git anyway. Not sure if you've read the discussions that were done on this topic. The Council decided (due to git and auto-generation of ChangeLogs) that ChangeLogs need to be amended, updated or changed. That to fix misleading typos, and more. ChangeLogs are meant for our users. For this reason repoman was changed to update the ChangeLog automatically on commit, if no changes to this file had been made. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH] emerge: --autounmask-write if --ask (bug 481578)
From: Alexander Berntsen alexan...@plaimi.net Signed-off-by: Alexander Berntsen berna...@gentoo.org --- For Brian to review, and anyone else who feels like it. man/emerge.1| 3 ++- pym/_emerge/depgraph.py | 7 +-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/man/emerge.1 b/man/emerge.1 index e32bb43..8bf3db1 100644 --- a/man/emerge.1 +++ b/man/emerge.1 @@ -367,7 +367,8 @@ to config files, respecting \fBCONFIG_PROTECT\fR and \fB\-\-ask\fR. If the corresponding package.* is a file, the changes are appended to it, if it is a directory, changes are written to the lexicographically last file. This way it is always ensured that the new changes take -precedence over existing changes. +precedence over existing changes. This option is automatically enabled with +\-\-ask. .TP .BR \-\-backtrack=COUNT Specifies an integer number of times to backtrack if diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py index cc87d9f..baa3129 100644 --- a/pym/_emerge/depgraph.py +++ b/pym/_emerge/depgraph.py @@ -7430,12 +7430,15 @@ class depgraph(object): (using CONFIG_PROTECT). The message includes the comments and the changes. - autounmask_write = self._frozen_config.myopts.get(--autounmask-write, n) == True + ask = --ask in self._frozen_config.myopts + autounmask_write_y = \ + self._frozen_config.myopts.get(--autounmask-write, y) + autounmask_write = not autounmask_write_y == n and \ + (ask or autounmask_write_y == True) autounmask_unrestricted_atoms = \ self._frozen_config.myopts.get(--autounmask-unrestricted-atoms, n) == True quiet = --quiet in self._frozen_config.myopts pretend = --pretend in self._frozen_config.myopts - ask = --ask in self._frozen_config.myopts enter_invalid = '--ask-enter-invalid' in self._frozen_config.myopts def check_if_latest(pkg): -- 1.8.5.5
Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific
Second patch containing only the ebuild.sh change.
Answers to suggestions inline.
On Thu, Sep 11, 2014 at 12:49 PM, Zac Medico zmed...@gentoo.org wrote:
On 09/11/2014 04:18 AM, Bertrand Simonnet wrote:
Hi guys,
I have been working on a patch to make package.env (and env/ files)
stackable with the profiles.
The main goal would be to have a mechanism to be able to define a
per-package, per-profile environment in a simple way.
We can currently do this with profile.bashrc but we have to add some
logic there to decide which package are relevant which already exist in
ebuild.sh and pym/*.
I attached my current patch which contain two main modifications:
- ebuild.sh walk the profile and source all scripts with path:
${profile_path}/env/${category}/${PN}
- config.py imports package.env from all profiles and then from
/etc/portage/package.env
It seems like duplication of work, to load the per-package env in
_grab_pkg_env on the python side, and to source it again on the bash
side. Why load the environments in both places? Note that the python
getconfig function only supports variable settings (no executable code),
which is very limited to what can be done when sourcing the files in bash.
Right.
The python part was mostly useful from a reuse point of view (write few
scripts
and enable them for some packages only. As the python mechanism is less
powerful
than I thought, I removed it.
Note:
* I gated the env/$CATEGORY/$PN scripts with a new feature:
per-profile-env.
I couldn't do this with package.env as the scripts are parsed before
we initialize self.features.
* I am not particularly attach to both package.env and
env/$CATEGORY/$PN. It might make more sense to have only
env/$CATEGORY/$PN, especially considering the problem with package.env.
Do you have any suggestions on this patch ?
1) If you want both the python-side env loading and the bash-side env
loading, I would make those into separate features, since sourcing the
files in bash supports full bash syntax while the python-side getconfig
function does not.
Fixed: removed the python part completly
2) When adding new global functions in ebuild.sh, please unset them
inside bin/save-ebuild-env.sh, when internal functions are stripped from
the environment.
Done.
3) You might want to look into using the repository's
metadata/layout.conf profile-formats setting, as an alternative to using
FEATURES to control profile-related behavior.
I replaced the FEATURES gate by a profile-formats option.
I added some logic to find layout.conf for a given profile path. (find the
first
parent dir named profiles then check ../metadata/layout.conf)
Is there a corner case that I missed?
Would you be willing to accept this patch once the few problems are
sorted out ?
I can't speak for everyone, but I'm pretty sure we can work something out.
--
Thanks,
Zac
Thanks,
Bertrand
From 8a7e493d343bd986fa14e3c7148e76012e6f98d6 Mon Sep 17 00:00:00 2001
From: Bertrand SIMONNET bsimon...@chromium.org
Date: Fri, 12 Sep 2014 05:07:20 -0700
Subject: [PATCH] Make env/ bash scripts profile specific
This generalize the /etc/portage/env mechanism to be stackable with profiles.
ebuild.sh will walk the profiles and sources scripts in env/ following the same
matching rules as for /etc/portage/env.
Change-Id: I16bcd75790213d2204f83d4aa6e8b910f8829b6e
---
bin/ebuild.sh| 78
bin/isolated-functions.sh| 23
bin/save-ebuild-env.sh | 1 +
pym/portage/repository/config.py | 2 +-
4 files changed, 73 insertions(+), 31 deletions(-)
diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index be044e0..87e44f4 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -67,11 +67,11 @@ unset BASH_ENV
# earlier portage versions do not detect a version change in this case
# ( to ) and therefore they try execute an incompatible version of
# ebuild.sh during the upgrade.
-export PORTAGE_BZIP2_COMMAND=${PORTAGE_BZIP2_COMMAND:-bzip2}
+export PORTAGE_BZIP2_COMMAND=${PORTAGE_BZIP2_COMMAND:-bzip2}
# These two functions wrap sourcing and calling respectively. At present they
# perform a qa check to make sure eclasses and ebuilds and profiles don't mess
-# with shell opts (shopts). Ebuilds/eclasses changing shopts should reset them
+# with shell opts (shopts). Ebuilds/eclasses changing shopts should reset them
# when they are done.
__qa_source() {
@@ -275,7 +275,7 @@ inherit() {
set +f
__qa_source $location || die died sourcing $location in inherit()
-
+
#turn off glob expansion
set -f
@@ -290,7 +290,7 @@ inherit() {
[ ${B_IUSE+set} = set ] IUSE=${B_IUSE}
[ ${B_IUSE+set} = set ] || unset IUSE
-
+
[ ${B_REQUIRED_USE+set} = set ] REQUIRED_USE=${B_REQUIRED_USE}
[ ${B_REQUIRED_USE+set} = set ] || unset REQUIRED_USE
@@ -372,42 +372,60 @@ __source_all_bashrcs() {
restore_IFS
for x in
Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific
On 09/15/2014 11:42 AM, Bertrand Simonnet wrote: I replaced the FEATURES gate by a profile-formats option. I added some logic to find layout.conf for a given profile path. (find the first parent dir named profiles then check ../metadata/layout.conf) Is there a corner case that I missed? I all looks good to me, except that I don't like the way that __profile_env_enabled parses the layout.conf for each directory, since it's terribly inefficient. It would be much better to parse the layout.conf data on the python side, and pass it to bash as an environment variable. For example, the PORTAGE_ECLASS_LOCATIONS variable is generated in python such that eval can be used to convert it to an array. We can add a similar array variable called PORTAGE_PROFILE_ATTRIBUTES, and each element of the array will correspond to the profile path at the same index in the path_array variable. -- Thanks, Zac
