Re: [gentoo-dev] Re: CIA replacement
On 02-10-2012 12:40:20 +0800, Ben de Groot wrote: > The irker proxy was mentioned in this thread. I think we should look > into this. Unless someone has a better idea. I think it might be more beneficial to try and relay -commits email to irc. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-dev] CVS -> git, list of where non-infra folk can contribute
Thank you so much for taking the time to give us this clear list of things that need to be done to take this forward! -- Cheers, Ben | yngwin Gentoo developer Gentoo Qt project lead, Gentoo Wiki admin
Re: [gentoo-dev] Re: CIA replacement
On 1 October 2012 17:48, Duncan <1i5t5.dun...@cox.net> wrote: > Ben de Groot posted on Mon, 01 Oct 2012 16:14:25 +0800 as excerpted: > >> Since CIA.vc is dead [1], I think we should be looking into a >> replacement service, or host our own [2]. >> Is infra already looking into this? >> >> 1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html 2: >> http://www.donarmstrong.com/posts/switching_to_kgb/ > > This has been discussed previously. > > Thread: CIA.VC down for the count? > Original post by blueness on August 23, last post on the 26th. > > > Here's a link to the gmane thread: > > http://comments.gmane.org/gmane.linux.gentoo.devel/79499 > > Old interface: > > http://thread.gmane.org/gmane.linux.gentoo.devel/79499 > > > It may be that there are further developments, but a read of the old > thread (which you don't appear to have participated in and didn't mention > so I'm assuming you missed) should help in any case. I didn't actually miss that thread, but in that case it was a short outage, and cia.vc came back the next day. People were pointed at the -commits ML, and that was it. Now that it's down for good (unless someone recreates it, but no news of that so far), we need a more permanent solution. The -commits ML is okay (tho I don't want to subscribe to such a high-volume ML), but we miss an IRC interface. The website and statistics of cia.vc were nice too. The irker proxy was mentioned in this thread. I think we should look into this. Unless someone has a better idea. (which is why I brought this up in the first place) -- Cheers, Ben | yngwin Gentoo developer Gentoo Qt project lead, Gentoo Wiki admin
[gentoo-dev] CVS -> git, list of where non-infra folk can contribute
Cross-posting to scm; responses should go to scm please (and the people who whinge about cross posting should go promptly to hell if I have any say in the matter). On Mon, Oct 01, 2012 at 05:58:43PM -0700, Diego Elio Petten?? wrote: > On 01/10/2012 17:51, Gregory M. Turner wrote: > > > > Anyhow, I get it: administering the vcs for a huge project such as > > Gentoo is very hard work. If I somehow gave some other impression, I'm > > sorry. Perhaps Rich and I insensitively voiced our shared assumption > > that Gentoo's continued reliance on cvs stems from a lack of motivation > > and consensus, rather than a shortage of labor and resources. > > That's definitely not the case. While we do have had some complains > (mostly from Prefix last I knew) about git's working, the consensus for > going to git is there. The problems are vastly technical. > > Problems such as "how many developers would be fine with having to > checkout 2GB of history to be able to commit"? git support shallow > clones but not if you want to commit to them. Few corrections; 1) You can commit to shallow clones. You can actually push from them too- you just have to know what you're doing (your parent *has* to be known to the other side, else you're trying to push a disconnected history/graph to the other side, which doesn't know how to connect the two). We won't be doing that fortunately, just noting that it is possible if you're careful (and I know what the man page says; what I'm saying is the full version, rather than the short version they list there). 2) graft's are what we'll be doing there; kind of shallow, but now. Basically the same thing the kernel folk did. As for the "quit your bitching and contribute already" rant angle; Diego's accurate; minimally, it's more productive to contribute and you're less likely to crap on folks motivation, let alone risk the wraith of a pissy person like me yelling at you. Here in is the kicker; certain chunks of this can't be handled by random joe blow off the street- they require core infra access. Bluntly (no disrespect to people, just being brutally direct) I don't care if you have infra friends, I don't care if you maintain a couple of boxes; if you're doing heavy OPs in a production environment, you'll understand the issue of trust/access- thus you'll understand that some of this work, cannot be done by anyone but infra. Like it or not, very few people have access to the core cvs -> rsync hosts/machinery- since each/every/one/of/us means it's a security angle that has to be tracked. That's not arguable, so don't even try please. That said, there are non-infra contributions people can make. I suggest people do that; here's the list off the top of my head (these are things worst case, I'll sort- which means it'll be months out till I finish them considering my own time constraints and focus on getting eapi5 support into pkgcore first). 0) First the rules of the road for this discussion; assume that I'll be bitchy if you violate this. 0.a) We're not dropping the existing history. Suggesting this is asking for a killfile entry, it's viable for small or throw-away projects; gentoo-x86 cvs repository is not a throw-away project. 0.b) Lesser offence since it's not obvious; the various suggestions that we just snapshot this, then try to fix history after the fact won't work- look into git's transitive trust via sha1's of the parent's sha1. To do that sort of proposal means forcing a full history rewrite down the line; this doesn't fly. 0.c) For whatever I've missed, assume that if it craps on developers workflow... it's a no go, and needs to be addressed. Does CVS suck? Yes, I hate having to use it. But it *works*; switching to git has to be, minimally, a lateral move for developers in terms of their workflow- we cannot make it worse else what's the point of this whole exercise? There may be an exception or two here- things that aren't sorted immediately upon conversion, but those exceptions will only fly if they're minor, don't require history rewrites, and someone is locked in/guranteed to be working on it now (else we have no gurantee it'll actually be sorted). 1) We need a thin manifest -> thick manifest converter. Thin manifests are used for git- they store just DIST entries. Thick (also known as 'full'), are what cvs/rsync users are familiar with- it holds checksums for all content. 1.a) This converter must use portage api's; ultimately, this thin->thick conversion will be signed by an infra key (rather than the current hodgepodge of devs). I suggest nesting it under the emaint command. 1.b) This converter needs to be fast. $VCS -> rsync updates occur every 30 minutes. thin/thick sorting should be sub minute, frankly; go parallel (multiprocessing) being my suggestion, threadpool worst case (since most of the work won't be gil bound). 1.c) This absolutely has to be fucking stable. This will be a core part
Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss
On Mon, Oct 1, 2012 at 8:58 PM, Diego Elio Pettenò wrote: > On 01/10/2012 17:51, Gregory M. Turner wrote: >> >> Anyhow, I get it: administering the vcs for a huge project such as >> Gentoo is very hard work. If I somehow gave some other impression, I'm >> sorry. Perhaps Rich and I insensitively voiced our shared assumption >> that Gentoo's continued reliance on cvs stems from a lack of motivation >> and consensus, rather than a shortage of labor and resources. > > That's definitely not the case. While we do have had some complains > (mostly from Prefix last I knew) about git's working, the consensus for > going to git is there. The problems are vastly technical. > > Problems such as "how many developers would be fine with having to > checkout 2GB of history to be able to commit"? git support shallow > clones but not if you want to commit to them. > >> Then >> again, if the folks in the trenches doing the work think I've slighted >> them, surely they are perfectly capable of chewing me out on their own >> behalfs and don't need you to do it for them? > > It is at the very least disturbing that you think that people whose > work, and commitment, has been overlooked for a whole discussion can't > be bothered by it unless they are actually reactive aggressively. > > I know the kind of thankless tasks Infra has to deal with on a daily > basis, and I think they deserve more respect that most of the time they > are given here, especially when technical challenges are billed under > the "we just need to push harder for it to move" banner like this time. Some fascinating problems (social and technical) to address. I joined the -scm list because it was implied that these things would be discussed over there. Could we "take this outside"? I'd be interested in looking at solutions...but I don't want to drop them in here, since it's been pounded on a few times by a few people that this isn't the place. -- :wq
Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss
On 01/10/2012 17:51, Gregory M. Turner wrote: > > Anyhow, I get it: administering the vcs for a huge project such as > Gentoo is very hard work. If I somehow gave some other impression, I'm > sorry. Perhaps Rich and I insensitively voiced our shared assumption > that Gentoo's continued reliance on cvs stems from a lack of motivation > and consensus, rather than a shortage of labor and resources. That's definitely not the case. While we do have had some complains (mostly from Prefix last I knew) about git's working, the consensus for going to git is there. The problems are vastly technical. Problems such as "how many developers would be fine with having to checkout 2GB of history to be able to commit"? git support shallow clones but not if you want to commit to them. > Then > again, if the folks in the trenches doing the work think I've slighted > them, surely they are perfectly capable of chewing me out on their own > behalfs and don't need you to do it for them? It is at the very least disturbing that you think that people whose work, and commitment, has been overlooked for a whole discussion can't be bothered by it unless they are actually reactive aggressively. I know the kind of thankless tasks Infra has to deal with on a daily basis, and I think they deserve more respect that most of the time they are given here, especially when technical challenges are billed under the "we just need to push harder for it to move" banner like this time. -- Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/
Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss
If you're going to paint me and the other folks expressing opinions as entitled mouth-breathers, certainly you can't expect not to hear any reply because it's "off-topic"! On 10/1/2012 2:00 PM, Diego Elio Pettenò wrote: On 01/10/2012 13:54, Rich Freeman wrote: I don't think we can keep the discussion off -dev forever though. It seems like we're close to being able to implement, which means lots of changes that impact all devs. I can't imagine that we'd want to implement that without some kind of council vote. Perhaps the appropriate approach is to propose a GLEP? No, the appropriate approach is first to _talk with Infra_. Guys I said that before, but unless you actually factor in Infra when you want infrastructure stuff done, you have to do the work yourself. And it might not be nice and fun. Well I definitely can't argue with the above, and I didn't know about -scm but, hey, you learn something new every day; anyhow I'm not sure it's makes it inappropriate to discuss. So to close this in a few words: You walk the walk, you talk the talk. And I have no intention to read another mail with "what other awesome thing we can do if we migrate to git and we don't even have to worry about what it might happen on the serverside because git is just magical and will sort itself out", okay? OK, also agreed that git lacks entirely in magical properties -- I wasn't aware of having said anything suggesting otherwise. I have some limited experience as a git admin and am aware of git's considerable limitations with respect to provisioning, security configuration, scalability and other non-optional features for a public-facing deployment. There is more to be said on the matter but I'll take that to -scm. Anyhow, I get it: administering the vcs for a huge project such as Gentoo is very hard work. If I somehow gave some other impression, I'm sorry. Perhaps Rich and I insensitively voiced our shared assumption that Gentoo's continued reliance on cvs stems from a lack of motivation and consensus, rather than a shortage of labor and resources. Then again, if the folks in the trenches doing the work think I've slighted them, surely they are perfectly capable of chewing me out on their own behalfs and don't need you to do it for them? I'm not looking for a fight, but after reading the above and some other remarks in this thread, by you and others, I did want to at least clarify my position: No, of course I couldn't possibly know all the repercussions of a change like this, but I also find it difficult to believe that whatever hurdles exist are so intractable that we'd might as well just throw in the towel. Although I regret any bad feelings I may have caused, I stand by my statements. To be clear, the "magic" of the Gentoo community -- not git -- is what I believe will make this doable, and yes, I appreciate that that "magic" is actually just a lot of people doing a lot of nasty, thankless chores. As for the whole put-up/shut-up business, I'd be happy to help out any way I can (although tbh I'd still say whatever was on my mind even if I was too busy). I'll contact Infrastructure to make sure they're aware of me. -gmt
Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss
On 01/10/2012 15:53, Peter Stuge wrote: >> > Then you probably don't know half the Gentoo developers > I think they are the ones who should fork. :) Unfortunately the problem is that they tend to linger around even after forking... -- Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/
Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss
Diego Elio Pettenò wrote: > > Another idea I have, besides the go-ahead+fix what breaks, is that > > after everything has broken, Gentoo developers will not be spamming > > this mailing list like three-year-olds screaming rude complaints > > about how things do not work and calling infra bad names, but that > > they will actually *help out* with whatever needs fixing. > > Then you probably don't know half the Gentoo developers I think they are the ones who should fork. :) //Peter
Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss
On 01/10/2012 15:21, Peter Stuge wrote: > Diego Elio Pettenò wrote: >> With all due respect, > .. >> you calling for shutdown dates > .. > all without changing a stupid subject > line to at least show you're no longer speaking about the original topic > (it's not like people can be psychic that you're talking about GIT > migrations when the topic says "CIA replacement"), Yes just cut out the part that makes it obnoxious, will ya? >> is obnoxious. > Another idea I have, besides the go-ahead+fix what breaks, is that > after everything has broken, Gentoo developers will not be spamming > this mailing list like three-year-olds screaming rude complaints > about how things do not work and calling infra bad names, but that > they will actually *help out* with whatever needs fixing. Then you probably don't know half the Gentoo developers -- Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/
Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss
Diego Elio Pettenò wrote: > With all due respect, .. > you calling for shutdown dates .. > is obnoxious. I don't know about respectful, but oh well.. Another idea I have, besides the go-ahead+fix what breaks, is that after everything has broken, Gentoo developers will not be spamming this mailing list like three-year-olds screaming rude complaints about how things do not work and calling infra bad names, but that they will actually *help out* with whatever needs fixing. Gentoo has a whole bunch of very competent developers in many different areas, including yourself of course!, and I'm pretty sure that there is no better way to get everything fixed *fast* than to simply go-ahead. Some (of course not all, and that's fine too) devs would surely get involved to help out with whatever issues need to be solved. //Peter
Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss
On 01/10/2012 14:41, Peter Stuge wrote: > Noone said it's easy. «This just strikes me as something that is about at the point where we could "just do it."» This was Rich at 11.29 Pacific Time. > Several said it needs to just-be-done, without further consensus. > I support that. Everyone also agrees that there will be issues, but > I think the idea is that switching sooner rather than later and > fixing up the bits that break is fine. Even if it takes a while. Okay so you have your idea. Keep it. But don't try to force it on people who are actual developers. > It shouldn't. There may be others on the list besides Gentoo infra > who know a thing or two about infrastructure, operations, CVS, Git, > and so on. Sure, but they are not Gentoo Infra. And who has to maintain this? Gentoo Infra. You want to do the conversion and maintain it yourself? Feel free, nobody's stopping you. You're free to fork. > I have no idea about that issue, but it seems quite distinct. It's the same bullshit of insisting that someone else should do something just because you think it's better, without actually asking for feasibility. > I could help out, I > know a couple of infra guys, but I can't even get recruited because > quizzes need too much contiguous time out of my schedule.. That's true for everybody. And knowing the infra guys doesn't mean that you're infra still. I'm not infra either and I don't speak _for_ them, but for this kind of stuff, instead of starting a SEVENTEEN POSTS thread on a mailing list that should be dedicated to other stuff, the solution is first _ask what the status is_. With all due respect, but having Michael going off a tangent on caching proxy servers, Rich starting to ponder between bandwidth and CPU, and you calling for shutdown dates, all without changing a stupid subject line to at least show you're no longer speaking about the original topic (it's not like people can be psychic that you're talking about GIT migrations when the topic says "CIA replacement"), is obnoxious. -- Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/
Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss
Diego Elio Pettenò wrote: > Honestly, this whole thread, with the exception of Rafael, makes me > facepalm incredibly, because everybody is saying "it's easy!" > without asking the people who have done the work up to now and will > have to manage it. Noone said it's easy. Please don't put words in my mouth. Several said it needs to just-be-done, without further consensus. I support that. Everyone also agrees that there will be issues, but I think the idea is that switching sooner rather than later and fixing up the bits that break is fine. Even if it takes a while. > And it pisses me off. It shouldn't. There may be others on the list besides Gentoo infra who know a thing or two about infrastructure, operations, CVS, Git, and so on. > most of the complains about the way tinderbox's logs are handled I have no idea about that issue, but it seems quite distinct. > So to close this in a few words: You walk the walk, you talk the talk. That's bullshit (and the saying is backwards). I could help out, I know a couple of infra guys, but I can't even get recruited because quizzes need too much contiguous time out of my schedule.. //Peter
[gentoo-dev] Discussing stuff that is not appropriate to discuss
On 01/10/2012 13:54, Rich Freeman wrote: > I don't think we can keep the discussion off -dev forever though. It > seems like we're close to being able to implement, which means lots of > changes that impact all devs. I can't imagine that we'd want to > implement that without some kind of council vote. Perhaps the > appropriate approach is to propose a GLEP? No, the appropriate approach is first to _talk with Infra_. Guys I said that before, but unless you actually factor in Infra when you want infrastructure stuff done, you have to do the work yourself. And it might not be nice and fun. Honestly, this whole thread, with the exception of Rafael, makes me facepalm incredibly, because everybody is saying "it's easy!" without asking the people who have done the work up to now and will have to manage it. And it pisses me off. (Among other things, because it feels like most of the complains about the way tinderbox's logs are handled, "it's easy!" but nobody but me is ever going to pick up the task, ...) So to close this in a few words: You walk the walk, you talk the talk. And I have no intention to read another mail with "what other awesome thing we can do if we migrate to git and we don't even have to worry about what it might happen on the serverside because git is just magical and will sort itself out", okay? -- Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/
Re: [gentoo-dev] CIA replacement
On Mon, Oct 1, 2012 at 4:35 PM, Diego Elio Pettenò wrote: > > And I suggest we stop here. We have a different mailing list for this > and it's getting tiring. Ok, looking at the archives as far as I can tell nobody is really monitoring that list (a post requesting a status update went unanswered sep 9th). So, rather than continuing to post here which seems to be annoying people, I'll just issue a call to anybody who cares to move discussion over to gentoo-scm. That said, I plan to mark any blockers to the migration as resolved where I don't see any remaining concerns unless somebody comments on -scm or on the relevant bugs that this isn't a good idea. I don't think we can keep the discussion off -dev forever though. It seems like we're close to being able to implement, which means lots of changes that impact all devs. I can't imagine that we'd want to implement that without some kind of council vote. Perhaps the appropriate approach is to propose a GLEP? Rich
Re: [gentoo-dev] CIA replacement
On 01/10/2012 13:17, Peter Stuge wrote: > Gregory M. Turner wrote: >> > "fuck everyone, we are doing this, here is the changeover date." > Well put. When is the date? I suggest October 5th, 18:00 UTC. And I suggest we stop here. We have a different mailing list for this and it's getting tiring. -- Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/
Re: [gentoo-dev] CIA replacement
On Mon, Oct 1, 2012 at 4:19 PM, Dirkjan Ochtman wrote: > On Mon, Oct 1, 2012 at 7:29 PM, Rich Freeman wrote: >> Why exactly are we still using cvs? Rather than building enhancements >> for cvs, why not just migrate everything to git, and spend our time >> building the git hooks/etc necessary to make this work? > > The CIA/irc bot issue is pretty much entirely orthogonal to the git > migration issues, so I don't think you should hijack this thread over > that. :) Yup, started a separate thread on -scm. My main point was that any issues with cvs are short-term issues. Do they really need to be fixed? Rich
Re: [gentoo-dev] CIA replacement
On Mon, Oct 1, 2012 at 7:29 PM, Rich Freeman wrote: > Why exactly are we still using cvs? Rather than building enhancements > for cvs, why not just migrate everything to git, and spend our time > building the git hooks/etc necessary to make this work? The CIA/irc bot issue is pretty much entirely orthogonal to the git migration issues, so I don't think you should hijack this thread over that. :) Cheers, Dirkjan
Re: [gentoo-dev] CIA replacement
Gregory M. Turner wrote: > "fuck everyone, we are doing this, here is the changeover date." Well put. When is the date? I suggest October 5th, 18:00 UTC. //Peter
Re: [gentoo-dev] CIA replacement
On 10/1/2012 10:29 AM, Rich Freeman wrote: Looking at the tracker [1], we need a pre-upload hook (I'm not quite sure why), an rsync conversion script, the ability to validate the converted tree, and documentation. There is still an open bug for commit signing, and I'm not quite sure why as this was implemented. a job for gitolite? what does ogo use? Am I missing some big concern that just isn't obvious in these bugs? Things will break. There will be much wringing of virtual hands. But in 1 weeks' time, everyone will get over it and find ways to fix whatever broke. There will be one less thing on the big to-do list, and most people will be happier overall. It's one of those "Social Security" type of things it's hard to build a clean consensus around /when/ to make a break like this. I think we basically need a headstrong asshole in a position of sufficient authority to say, "fuck everyone, we are doing this, here is the changeover date." Too much democracy, tactical planning and consensus-building will almost certainly perpetuate the status quo. Look, Gentoo will probably fall to pieces for a few days. The alternative, IMO, is a huge, elaborate multi-phase plan of some kind, like how Microsoft or NASA would do it: armies of 40-somethings toil for months or years in cubicles... Elaborate Gantt charts and process diagrams are drawn up and approved by layer after layer of middle management. Gentoo can tolerate a hiccup; it's survived way scarier challenges. We have a huge community of very talented people highly motivated to keep this machinery working... there's literally zero chance that Gentoo will somehow have it's spirit broken by cvs->git. In other words, devs -- imo -- just go for it. Throw caution to the wind. Break some bylaws if need be. Ignore anyone saying "we're not ready." It'll be easier to get forgiveness than permission, and exactly what process issues needed to be smoothed over will frankly only truly be clear in the aftermath. Of course, that's easy for me to say since I'm not the one whose door everyone will be beating down when TSHTF. There are worse things than CVS (Visual SourceSafe comes to mind). But if we don't bite the bullet now, we all know we'll be having this discussion again later. -gmt
Re: [gentoo-dev] CIA replacement
Rich Freeman wrote: > I doubt everybody is going to be happy if somebody convinces infra > to shut down cvs without any discussion first. I would do exactly that, actually. There's been years of discussion. There's even a mailing list for discussion. //Peter
Re: [gentoo-dev] CIA replacement
On 01/10/2012 11:29, Rich Freeman wrote: > Where else would one discuss it? gentoo-scm Yes, there is a mailing list for that. -- Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/
Re: [gentoo-dev] CIA replacement
On Mon, Oct 1, 2012 at 2:08 PM, Rafael Goncalves Martins wrote: > Have you ever thought that people may be not really interested on this > move? or don't have the time to work on it? or don't care enough to > spend time on it? or just wants someone else to do the work? > I'd thought of every single one of those. Hence my post on the list to drum up interest. > If you want it, go ahead and push it, but in the right places, please. > I'm tired of bikeshedding here in this list. :( Where else would one discuss it? Certainly happy to have a separate thread - it just seemed like an obvious question when somebody brought up making an enhancement to our cvs infrastructure. I have no interest in posting this on a list that nobody actually reads, but if there are a bunch of devs following some other list that is more appropriate by all means let me know. This just strikes me as something that is about at the point where we could "just do it." Obviously the discussion needs to involve at least some core of developers, since the goal is to get everybody to stop doing their commits in cvs, and start doing them in git, and I doubt everybody is going to be happy if somebody convinces infra to shut down cvs without any discussion first. Rich
Re: [gentoo-dev] CIA replacement
On Mon, Oct 1, 2012 at 1:54 PM, Rich Freeman wrote: > On Mon, Oct 1, 2012 at 1:42 PM, Michael Mol wrote: >> I don't know to what depth this has been discussed in the past, but if >> you use git, you also get an HTTP transport, which has a useful >> feature: You could simplify updating the tree on end-users's machines >> by using caching proxy servers (operating in accelerator mode) on the >> various mirrors. > > The issue I see here is a tradeoff of bandwidth vs CPU. I just ran an > emerge --sync and the total amount of transmitted data was 5M. The > whole tree is 250M, though no doubt with compression that could be > reduced. > > Now, one advantage of HTTP is that caching http servers are likely > more ubiquitous in general than rsync servers. But, we have a whole > bunch of rsync servers already, and we don't have a bunch of caching > http servers. > > I suspect bandwidth is going to cost more than CPU here. > > In any case, not a reason to hold up git, just one more possibility if > we ever move. It really depends on how efficient 'git pull' is over HTTP, IMO. I mean, when I do pulls and pushes in my workflows, it doesn't do a full data send or a full data pull; that's reserved for 'git clone'. It may also depend on how often the pull is done; rsync doesn't pull a tree history, just a copy at the time of sync. git pulls new objects, which may include intermediate versions which are no longer necessary. (git may be capable of only syncing to 'HEAD' without worrying about intermediate states, but I don't know. I'm not advanced enough to definitively say one way or another.) As for setting up caching proxies at existing mirror sites...It should be a case of publishing, a single squid (or pick whatever proxy you prefer to standardize on) configuration file with appropriate ACLs and copy it all over the place. I'm not on the infra team, so I don't know what things look like under the hood, but I do know setting up squid in forward and reverse roles isn't very painful. -- :wq
Re: [gentoo-dev] CIA replacement
On Mon, Oct 1, 2012 at 2:29 PM, Rich Freeman wrote: > On Mon, Oct 1, 2012 at 11:21 AM, Rafael Goncalves Martins > wrote: >> Maybe someone with good cvs knowledge can contribute a hook for irker >> [1], so we can have #gentoo-commits flooding our irc clients again! :) > > Why exactly are we still using cvs? Rather than building enhancements > for cvs, why not just migrate everything to git, and spend our time > building the git hooks/etc necessary to make this work? It is amazing how a simple thread about a quite simple topic, with an easy solution, like this, turns to useless discussions about endless topics, like this git conversion. This mailing-list is getting really boring. :( I'm not asking nobody to write any "enhancement for cvs". irker is already "done", and working fine, but the authors just implemented support for git and svn, because this is what they use. We *use* CVS for now, then we would need to fix it to work with CVS, no matter if we will have git repositories at some time. I stopped believing in Santa Claus when I was 5 years old. This should be doable with 10 lines of python, or using the cia->irker proxy that jdhore mentioned in another email. > Looking at the tracker [1], we need a pre-upload hook (I'm not quite > sure why), an rsync conversion script, the ability to validate the > converted tree, and documentation. There is still an open bug for > commit signing, and I'm not quite sure why as this was implemented. Have you ever thought that people may be not really interested on this move? or don't have the time to work on it? or don't care enough to spend time on it? or just wants someone else to do the work? If you want it, go ahead and push it, but in the right places, please. I'm tired of bikeshedding here in this list. :( [snip] Best regards, -- Rafael Goncalves Martins Gentoo Linux developer http://rafaelmartins.eng.br/
Re: [gentoo-dev] CIA replacement
On Mon, Oct 1, 2012 at 1:42 PM, Michael Mol wrote: > I don't know to what depth this has been discussed in the past, but if > you use git, you also get an HTTP transport, which has a useful > feature: You could simplify updating the tree on end-users's machines > by using caching proxy servers (operating in accelerator mode) on the > various mirrors. The issue I see here is a tradeoff of bandwidth vs CPU. I just ran an emerge --sync and the total amount of transmitted data was 5M. The whole tree is 250M, though no doubt with compression that could be reduced. Now, one advantage of HTTP is that caching http servers are likely more ubiquitous in general than rsync servers. But, we have a whole bunch of rsync servers already, and we don't have a bunch of caching http servers. I suspect bandwidth is going to cost more than CPU here. In any case, not a reason to hold up git, just one more possibility if we ever move. Rich
Re: [gentoo-dev] CIA replacement
Well nenolod has written a CIA -> Irker proxy that (I believe) takes commit messages designed to go to CIA and makes irker read them and such, but i haven't looked into it: https://github.com/nenolod/irker-cia-proxy On 1 October 2012 11:21, Rafael Goncalves Martins wrote: > Hi Ben, > > On Mon, Oct 1, 2012 at 5:14 AM, Ben de Groot wrote: >> Since CIA.vc is dead [1], I think we should be looking into a >> replacement service, or host our own [2]. >> Is infra already looking into this? >> >> 1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html >> 2: http://www.donarmstrong.com/posts/switching_to_kgb/ > > Maybe someone with good cvs knowledge can contribute a hook for irker > [1], so we can have #gentoo-commits flooding our irc clients again! :) > > [1] http://www.catb.org/esr/irker/ > > Best regards. > > -- > Rafael Goncalves Martins > Gentoo Linux developer > http://rafaelmartins.eng.br/ >
Re: [gentoo-dev] CIA replacement
On Mon, Oct 1, 2012 at 1:29 PM, Rich Freeman wrote: > On Mon, Oct 1, 2012 at 11:21 AM, Rafael Goncalves Martins > wrote: >> Maybe someone with good cvs knowledge can contribute a hook for irker >> [1], so we can have #gentoo-commits flooding our irc clients again! :) > > Why exactly are we still using cvs? Rather than building enhancements > for cvs, why not just migrate everything to git, and spend our time > building the git hooks/etc necessary to make this work? > > Looking at the tracker [1], we need a pre-upload hook (I'm not quite > sure why), an rsync conversion script, the ability to validate the > converted tree, and documentation. There is still an open bug for > commit signing, and I'm not quite sure why as this was implemented. > > It seems like a lot has already been done with validation. Checking > the active tree is pretty trivial - just compare the trees and they > should be the same. I guess we need to check history, but it seems to > me like the risk of problems is low, and if we just keep a backup of > the cvs repository if there is ever a concern about who made some > commit 5 years ago we can always dig it up. > > It really seems to me like little remains to be done here. Mostly we > just need somebody to push a decision on things like workflow. A few > of the bugs have comments like "no sense working on this with other > stuff still needed" - which seems to be outdated thinking with so > little left to do. > > Am I missing some big concern that just isn't obvious in these bugs? > > I also fear that we're refusing to take action on a great solution > because it isn't a perfect solution. Nobody in the world is using > tree-signing with git, and we aren't really using it in cvs either. > We now have the ability to do it with git, but depending on workflow > 3rd-party signatures might not end up in the history of head, or we > might not be able to verify them in an automated fashion. Honestly, I > think the appropriate response here is whoop-de-doo. We can't do any > of that stuff with cvs, but moving to git would have a lot of other > benefits. We can always change our processes later once somebody has > a solution for the signing problem. Right now we're making do without > it on cvs, and so is every other project using git. We can also > continue to sign manifests as a workaround, which is what we'll be > doing anyway if we never migrate to git. > > The git migration just strikes me as one of those cases where anybody > is free to come up with a reason not to use something, but nobody has > to defend keeping the status quo. I think the question isn't whether > there is anything wrong with using git, but whether the problems with > git are worse than the problems we already have. > > But, hey, if somebody wants to write an irc bot that posts cvs > commits, knock yourself out. > > Rich > > [1] - https://bugs.gentoo.org/show_bug.cgi?id=333531 > I don't know to what depth this has been discussed in the past, but if you use git, you also get an HTTP transport, which has a useful feature: You could simplify updating the tree on end-users's machines by using caching proxy servers (operating in accelerator mode) on the various mirrors. Those of us who have our own, local caching proxy servers (I have squid running on my network gateway) can reduce the network load even further by getting cache hits on our local network before even making queries outside our network. (Personally, I find this a far easier thing to maintain and do debugging reasoning on than, e.g. sharing a network mount or running a local rsync server managed by a cron job.) -- :wq
Re: [gentoo-dev] CIA replacement
On Mon, Oct 1, 2012 at 11:21 AM, Rafael Goncalves Martins wrote: > Maybe someone with good cvs knowledge can contribute a hook for irker > [1], so we can have #gentoo-commits flooding our irc clients again! :) Why exactly are we still using cvs? Rather than building enhancements for cvs, why not just migrate everything to git, and spend our time building the git hooks/etc necessary to make this work? Looking at the tracker [1], we need a pre-upload hook (I'm not quite sure why), an rsync conversion script, the ability to validate the converted tree, and documentation. There is still an open bug for commit signing, and I'm not quite sure why as this was implemented. It seems like a lot has already been done with validation. Checking the active tree is pretty trivial - just compare the trees and they should be the same. I guess we need to check history, but it seems to me like the risk of problems is low, and if we just keep a backup of the cvs repository if there is ever a concern about who made some commit 5 years ago we can always dig it up. It really seems to me like little remains to be done here. Mostly we just need somebody to push a decision on things like workflow. A few of the bugs have comments like "no sense working on this with other stuff still needed" - which seems to be outdated thinking with so little left to do. Am I missing some big concern that just isn't obvious in these bugs? I also fear that we're refusing to take action on a great solution because it isn't a perfect solution. Nobody in the world is using tree-signing with git, and we aren't really using it in cvs either. We now have the ability to do it with git, but depending on workflow 3rd-party signatures might not end up in the history of head, or we might not be able to verify them in an automated fashion. Honestly, I think the appropriate response here is whoop-de-doo. We can't do any of that stuff with cvs, but moving to git would have a lot of other benefits. We can always change our processes later once somebody has a solution for the signing problem. Right now we're making do without it on cvs, and so is every other project using git. We can also continue to sign manifests as a workaround, which is what we'll be doing anyway if we never migrate to git. The git migration just strikes me as one of those cases where anybody is free to come up with a reason not to use something, but nobody has to defend keeping the status quo. I think the question isn't whether there is anything wrong with using git, but whether the problems with git are worse than the problems we already have. But, hey, if somebody wants to write an irc bot that posts cvs commits, knock yourself out. Rich [1] - https://bugs.gentoo.org/show_bug.cgi?id=333531
Re: [gentoo-dev] [RFC] Initial python-r1.eclass & distutils-r1.eclass
On 01-10-2012 17:17:40 +0200, Marien Zwart wrote: > There used to be a more subtle problem: the python objects you import > have the path to their source embedded into them, and if you move the > .pyc/.pyo file around that path would be wrong, confusing some (mostly > debugging) tools (see http://bugs.python.org/issue1180193 ). This was > fixed fairly recently (in python 3, python 2.7 and python 2.6.2), so > it is probably no longer worth worrying about, but it's probably one > reason these files were handled this way in the past. This most likely is the reason why my tests with Python 2.7 worked fine when I tinkered with the .pyc and/or moved it around. Thanks -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-dev] CIA replacement
Hi Ben, On Mon, Oct 1, 2012 at 5:14 AM, Ben de Groot wrote: > Since CIA.vc is dead [1], I think we should be looking into a > replacement service, or host our own [2]. > Is infra already looking into this? > > 1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html > 2: http://www.donarmstrong.com/posts/switching_to_kgb/ Maybe someone with good cvs knowledge can contribute a hook for irker [1], so we can have #gentoo-commits flooding our irc clients again! :) [1] http://www.catb.org/esr/irker/ Best regards. -- Rafael Goncalves Martins Gentoo Linux developer http://rafaelmartins.eng.br/
Re: [gentoo-dev] [RFC] Initial python-r1.eclass & distutils-r1.eclass
(pardon any awkward formatting, writing from webinterface instead of a "proper" client. Please yell at me off-list or on irc if this post is excessively broken.) > https://bugs.gentoo.org/show_bug.cgi?id=300922 >From that bug: > - chpathtool does some "simple" heuristic to switch the build EPREFIX > to the new target EPREFIX > > - python (seemingly) doesn't like when the pyc files are changed and *not* > regenerated. > You will see the message: "Bad marshalling data" I'm not sure if I'm interpreting this correctly, but it sounds as if your chpathtool edits the .pyc file to replace the path? If yes: that will indeed not work, as strings are stored length-prefixed. You cannot sensibly edit .pyc files, you should normally just rewrite them. The .pyc/.pyo file store a version-dependent magic, the mtime of their corresponding .py file, and (since recently) the size of the correspondig .py file. If either changes the .pyc/.pyo is regenerated. The magic does not normally change in released pythons, so that's not much of a problem. The mtime is probably no longer a problem these days either. There used to be a more subtle problem: the python objects you import have the path to their source embedded into them, and if you move the .pyc/.pyo file around that path would be wrong, confusing some (mostly debugging) tools (see http://bugs.python.org/issue1180193 ). This was fixed fairly recently (in python 3, python 2.7 and python 2.6.2), so it is probably no longer worth worrying about, but it's probably one reason these files were handled this way in the past. -- Marien Zwart (marienz)
Re: [gentoo-dev] making USE=upnp a global flag
On Sun, 2012-09-30 at 17:44 +0200, Gilles Dartiguelongue wrote: > Le mercredi 19 septembre 2012 à 10:19 +0200, Michał Górny a écrit : > > > > Just to make it clear: > > - USE=upnp for upnp-igd or nat-pmp, > > - USE=dlna for the video magic and so on. > > > > Do I understand correctly? > > No, TV makers and others advertise UPnP as DLNA (digital living network > appliance) but that actually refers to both port forwarding (eg. used in > consoles) and to media streaming (eg. PC to TV). DLNA is actually a subset of UPnP with some strict certification rules. -- Olivier Crête tes...@gentoo.org Gentoo Developer signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] CIA replacement
Ben de Groot schrieb: > Since CIA.vc is dead [1], I think we should be looking into a > replacement service, or host our own [2]. I understand that ohloh is already tracking us (sometimes at least). http://www.ohloh.net/p/gentoo/ Best regards, Chí-Thanh Christopher Nguyễn
[gentoo-dev] Re: CIA replacement
Ben de Groot posted on Mon, 01 Oct 2012 16:14:25 +0800 as excerpted: > Since CIA.vc is dead [1], I think we should be looking into a > replacement service, or host our own [2]. > Is infra already looking into this? > > 1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html 2: > http://www.donarmstrong.com/posts/switching_to_kgb/ This has been discussed previously. Thread: CIA.VC down for the count? Original post by blueness on August 23, last post on the 26th. Here's a link to the gmane thread: http://comments.gmane.org/gmane.linux.gentoo.devel/79499 Old interface: http://thread.gmane.org/gmane.linux.gentoo.devel/79499 It may be that there are further developments, but a read of the old thread (which you don't appear to have participated in and didn't mention so I'm assuming you missed) should help in any case. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
[gentoo-dev] Re: [gentoo-pms] GLEP: gentoo sync based unified deps proposal
On Mon, 1 Oct 2012 02:01:32 -0700 Brian Harring wrote: > On Mon, Oct 01, 2012 at 08:13:49AM +0100, Ciaran McCreesh wrote: > > x? ( build: a run: b ) *is* nested "conflicting". > > > > You're still failing to understand the point of labels parsing > > rules, though: the point is to make uses like the above well > > defined and consistent. > > I understand them just fine; you're just either very fucking daft, > which I have a hard time believing, or lieing through your teeth > (which fits a decade of behaviour including multiple suspensions for > exactly that behaviour). > > Implicit labels context is build+run. Meaning the following > > x? ( build: a run: b ) *is* nested "conflicting". > > is actually > > build+run x? ( build: a run: b ) > > Which isn't a nested conflict- subset, not conflict. As I said right at the start, you're special-casing the top level to something that can't normally be expressed using the syntax. > You argue labels are required so people can do nested conflicts; > meaning the following extreme example: > > run x? ( build: a test: b ) > > And as I nicely pointed out, /not a single fucking exheres/ does > that. you've yet to pull out an example contradicting that analysis > in addition. No, I argue that having well-defined parsing rules means it doesn't matter if someone does do that. Meaning, no special case for the top level. Your rules require a handler to say "have I seen any dep: blocks further up the tree than my current position? If yes, handle this dep: block one way; otherwise, handle it a different way". With labels, all you do is initialise the label stack with build+run, and then no special case handling is required. That's what you should be putting in the GLEP. Not examples, but a big fat warning that your syntax requires a very strange special case rule to handle your default build+run behaviour. > What I truly love about that solution there is that it's both > accurate, and if I play my cards right, I may be able to get a glep > passed calling your proposal academic wankery; minimally, it'll be > fun from my standpoint to try, so at least something came out of the > last few emails from you. Oh come on, we all know that unnecessarily screwing up the syntax won't make DEPENDENCIES be sufficiently un-exherbo-looking to get it passed... -- Ciaran McCreesh signature.asc Description: PGP signature
[gentoo-dev] Re: [gentoo-pms] GLEP: gentoo sync based unified deps proposal
On Mon, Oct 01, 2012 at 08:13:49AM +0100, Ciaran McCreesh wrote: > x? ( build: a run: b ) *is* nested "conflicting". > > You're still failing to understand the point of labels parsing rules, > though: the point is to make uses like the above well defined and > consistent. I understand them just fine; you're just either very fucking daft, which I have a hard time believing, or lieing through your teeth (which fits a decade of behaviour including multiple suspensions for exactly that behaviour). Implicit labels context is build+run. Meaning the following > x? ( build: a run: b ) *is* nested "conflicting". is actually build+run x? ( build: a run: b ) Which isn't a nested conflict- subset, not conflict. You argue labels are required so people can do nested conflicts; meaning the following extreme example: run x? ( build: a test: b ) And as I nicely pointed out, /not a single fucking exheres/ does that. you've yet to pull out an example contradicting that analysis in addition. So... with that in mind- I'm doing two things; 1) can't force you back under a bridge, instead I'll do the killfile equivalent for a few weeks, 2) my original proposal if you kept being a tool seems appropriate: """ As said, you come up w/ real world examples, I'll include them; else persist and I'll just fold the academic wankery description of labels into the glep if you'd truly like me to (or you piss me off enough I do so to be a dick). """ What I truly love about that solution there is that it's both accurate, and if I play my cards right, I may be able to get a glep passed calling your proposal academic wankery; minimally, it'll be fun from my standpoint to try, so at least something came out of the last few emails from you. hugs and kisses- ~harring
[gentoo-dev] CIA replacement
Since CIA.vc is dead [1], I think we should be looking into a replacement service, or host our own [2]. Is infra already looking into this? 1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html 2: http://www.donarmstrong.com/posts/switching_to_kgb/ -- Cheers, Ben | yngwin Gentoo developer Gentoo Qt project lead, Gentoo Wiki admin
Re: [gentoo-dev] [RFC] Initial python-r1.eclass & distutils-r1.eclass
On Mon, Oct 01, 2012 at 08:36:12AM +0200, Fabian Groffen wrote: > On 30-09-2012 14:47:17 -0700, Brian Harring wrote: > > > In the worst case it returns "Bad marshalling data". > > > > Examples wanted for this. If this occurs, that's a python bug- one > > exception... portage (figures). They install into a non > > /usr/lib/python* location, meaning the .pyc/.pyo from py2.6 is > > exposed/accessed for py2.7 for example. > > https://bugs.gentoo.org/show_bug.cgi?id=300922 > > I doubt whether it's a Python bug, we have to mess with the files. But > then again, I did some toying, and it seems Python doesn't care about > this (any more?). Well, offhand that bug is pre EAPI3 (eapi3 was approved 01/18/10, and adoption was slow- lot of people skipped straight to eapi4) - so the mtime wouldn't have been guaranteed preserved for a long while. Meaning the bugs data I don't trust to be relevant due to timing, and age. As you said, this needs revisiting- minimally, portage is screwing around contents there, and I don't trust the python eclass to /not/ be forcing a compileall after the fact anyways. Suggest backing down the various protections for a full test, and resuming that bug- if you can replicate it, I'm definitely interested (dealt with this when it occurred for 2.3->2.4 for example). ~harring
[gentoo-dev] Re: [gentoo-pms] GLEP: gentoo sync based unified deps proposal
On Sun, 30 Sep 2012 16:56:56 -0700 Brian Harring wrote: > On Sun, Sep 30, 2012 at 10:53:40PM +0100, Ciaran McCreesh wrote: > > But here's the thing: when you sell something as "pragmatic", what > > you're really saying is "it's wrong, I know it's wrong, and I'm > > going to pretend that wrong is a good thing". Getting it wrong > > should be something you do only after you're sure you can't afford > > get it right; it shouldn't be something you're proud of. > > No, when I say pragmatic, what I'm actually saying is that people who > can't focus on cost/gain, by large, haven't had real jobs (else they > would've had that perfectionism/decreasing gains ground out of them > sooner or later), and are spending their time whacking off chasing a > mythical 'perfect' solution. I don't know whether you're aware of this, but a small number (cost per ebuild) multiplied by a big number (lots of ebuilds) can be larger than a medium sized number (cost of implementing a good solution). I realise this is a sophisticated technique I'm using here, but I assure you multiplication has been used in some industries for a few years now. > Academic wankery, is the short version. You're good at technical, > but you frequently do the academic wanking crap which leads to things > dead-ending... plus wasted time because to you, 'pragmatic' is a > dirty word (compromise? Heaven forbid!). Or looking at it another way: you're so eager to deliver a "compromise" and a "pragmatic" solution (at the expense of ebuild writers everywhere) that you immediately rule out a "good" solution just so you can push the virtues of doing it wrong. Doing it wrong should be a last resort, not something you look to do at any given opportunity. > > > In my proposal, I am addressing labels; will fold in your claims, > > > but those claims basically are shit- however, if you *did* find a > > > conflicting nested example that wasn't contrived, preferablly > > > multiple, I'd like those examples so I can include them into the > > > proposal (give labels a fair hand, basically). > > > > You already have an example in your proposal, in the form of > > mplayer's X? ( ) dependencies. > > I said nested conflicting labels. Meaning > "build: x? ( dar run: blah )" > > which isn't the case for any of mplayer deps. x? ( build: a run: b ) *is* nested "conflicting". You're still failing to understand the point of labels parsing rules, though: the point is to make uses like the above well defined and consistent. > We are not exherbo- we do not have the luxury of chucking out syntax > and pulling NIH renaming of things for shits and giggles. Especially > if the new syntax is directly translatable into a tweak of our > existing syntax (a tweak that we should do anyways- recall I built > this off of fixing USE_EXPAND). This isn't chucking out syntax. It's augmenting existing syntax. What you're doing is trying to shove something new onto an existing syntax which doesn't fit it. You should know this from REQUIRED_USE: that's a perfect example of going too far to reuse existing syntax. > Your argument boils down to "it's not labels, ignore that it's > aesthetic knob polishing (you can do the same w/ our existent > syntax, thus the analogy of waxing it I truly mean), use labels > because I'll berate you incessently till you accede". It's about giving ebuild developers a good format to work with. That sort of thing matters. There are a lot of ebuilds and not many package manglers. -- Ciaran McCreesh signature.asc Description: PGP signature
