On Mon, Dec 05, 2016 at 10:13:05PM +0100, Philip Hands wrote: > Ron <r...@debian.org> writes: > > > I'm not insisting that's what we should do. But it's certainly an > > option, and it dodges the bullet of having to say "Sucks to be you" > > without any notice at all. And it doesn't take anything away from > > the people who want "new upstream or bust" for Stretch, because it > > can still be available to them in backports. > > Perhaps you'd be kind enough to either confirm or correct my perceptions > of the current situation:
I'll try to be concise, but I might fail at 'brief', since I think it's important to fairly summarise all sides of this. Leaving people to somehow reconcile narrowly polarised, conflicting sound bites isn't very helpful. And since I don't have the option to abstain from an opinion, and aren't starting from an immutable one, I want to be as sure as I can that _I've_ got the whole picture clear in my head before settling on what I think is really best too. If I show how I got there, and you disagree with my rationale, then I have something that _I_ can rethink to maybe agree with you. If all I do is state a conclusion, and handwave the details of how I got there, then at best we can agree to disagree, and at worst we get more random trolls throwing mindless insults at me again. > Version 6 includes a CGI script that one is expected to install in a > manner so hopelessly insecure that we'd not accept it in Debian. For the version that Punit and Vincent wanted me to let them upload and that people complained that I nacked, which is where this appeal to the ctte started from, that's absolutely true. Not only did it have the 'chmod 777' interface to enable it, it had little gems in it like this too: open(PIPE, '@globalpath@' . " --result=ctags-xid $flags $pattern |"); Which for those who don't speak it, is perl for "anyone can execute arbitrary shell commands by typing them into a web browser", since $pattern is an unsanitised, untrusted, input from the query string. The version Punit published and suggested people should use had all of that enabled in it - and he did that after I'd warned him there was trouble that shouldn't just be ignored. That was the shared state and history when this ctte bug was opened. In the interests of seeing what new options we had _today_, I then did an initial review of the most recent upstream state, which nobody else had yet looked at, to see if anything material had changed which might improve the situation. My second post here included details of that, but the short answer is "it got more complicated". That one had removed the ability to run it from a secure system CGI location at all, and changed the way that it's generated yet again. It also made changes that guarantee everyone will need to fork it for distro use, because it now hardcodes a fun mix of paths, like /opt/local/bin for perl and /usr/local for global et al. It does comment out that line from above with a note: "To use this code, please uncomment in your own responsibility." But it also outputs a .htaccess enabling execution in the directory where the output is generated, whether you want to use it from there or not (and adds a second CGI, and a bunch of jquery doing AJAX too). It has a bunch of immediately obvious problems: - it still passes the unsanitised $pattern to global, which then passes it to popen (which again lets it be interpreted by the shell). - it won't run with perl's taint mode enabled, because that simply kills it warning of unsafe operations on untrusted data. - perl's warnings aren't enabled, and it complains of other rookie code problems if you do. - Enabling 'use strict' makes it scream with pain and completely fail to compile and run. It would need more thorough auditing to confirm that there is(n't) an exploitable path through that, but given that it ignores even the most basic advice which perl has strongly recommended since perl4 was new and shiny, then if there isn't, it would be because of luck more than obvious care. I'd not think it was a Good Plan to push this out as a "rush to beat the freeze" upload of this source without a much more careful look at it, and some effort spent actually fixing the already obvious deficiencies, but the size of the truck you might drive through its holes is a bit different to where we started from. It might not be utterly unfixable, but you'd have to convince upstream that security is important, or completely fork it for debian, which brings us back to exactly where we are - unless we just remove it all. But that would need Time, whoever does it. It is what we now have enabled in the package that Wookey uploaded to experimental. > However, it is possible to generate static content ... Yes. > ... that achieves the > same aims, but at the cost of approximately doubling the disk usage, > and presumably also requiring more time to generate. No. This isn't just a time-space tradeoff for equivalent functionality as Punit claimed. You lose the ability to actually search the code, which is the whole point of doing this in/with global in the first place. If you don't have that, you can get much better results by using doxygen to do the formatting, even if you don't have any doxygen specific markup in your source at all, and it will also give you client-side searchability (which these days might well be better than global's is anyway) ... So the options for global6 really boil down to: - Do we try to fork/fix the (new) existing CGI, to make it at least vaguely secure, modulo not being able to actually install it to a secure system CGI location. - Do we try to fork/fix htags to remove that, leaving it with just (IMO) relatively useless (compared to other options) functionality to create static markup. - Do we just remove htags completely, and only ship the tagging part of global (for Stretch at least, until someone does fix the problems with it). > Also, for people that want personal access to htags there is a > htags-server command that brings up a dedicated server to run the CGI > as the invoking user, by default bound to a localhost port. htags-server is a shell script that generates python or ruby code to use the HTTP server functionality they can provide. I have no idea offhand what the security properties of either of those is is really like, but AIUI the python version at least claims to run the CGIs as 'nobody' - but I'm not sure how that works when run as a user unless something is SUID somewhere ... or if that's patched to work differently in Debian to what the python docs say it does. If that's true it would mean you need everything to be at least world readable to use this. It still unconditionally drops a .htaccess file into the output though, so if you're running a real web server, that may or may not grant CGI execute access to that directory, depending on how it's otherwise configured - whether you use htags-server for this or not. That script doesn't really change any of the things that we'd be concerned about in practice. It just means people who don't have apache (or another web server) installed can install python or ruby instead. Either can limit it to localhost or allow access on a public IP. > Version 6 fixes some bugs that are still present in your version 5 > package and/or provides features that are missing, but bug reports of > sufficient quality to allow you to fix/backport to v5 are lacking. There appears to be two main issues that we 'know' using v6 'fixes'. One is whatever it is that the third-party ggtags wrapper needs, which aiui is what Vincent and Punit are most annoyed about. But I don't use emacs, and ggtags isn't even in Debian - and they haven't even told me what error they see, let alone what operation(s) trigger it. Vincent just gave me the output of global's short --help, and said "look, there's new options" - but we don't even know if it's actually a 'missing' command line option that it fails on (or which one that might be), or something else entirely. My hunch is that one would probably be pretty trivial to fix - either in ggtags or global, or both - if someone who uses it engages with what I asked originally, to file a separate bug from the 'new upstream' one, detailing the actual problem, and is willing to test any proposed fixes even if they aren't up to actually submitting a patch for that themselves. I don't mind writing a patch if we know what actually needs patching. Or if that really did prove to be intractable to backport, we'd have a real data point for the question we're looking at here too. But right now, the odds are good that it's something quite minor, because not that much has really changed. The other one, Wookey has now given us a good report for as part of his recent digging - and at first blush it appears to be 'trivial' in the sense that it doesn't actually appear to be a bug in global's code, but rather a hard limit in the version of flex that was used to generate the scanner. (I do consider it a bug in global that it isn't regenerated at build time - upstream doesn't even provide makefile rules to do it, they generate it 'manually' and bundle the generated file in the source). That one is only still open because it's going to be an intrusive patch, even if it's 'easy', and he wasn't keen to try regenerating it himself, and by that point I wasn't sure if it was going to just be wasted effort, if the consensus was going to be to kill off v5 right now. I've got a bunch of things I want to get done that I do expect to be released, but I can fit this one onto that priority list if we do decide that keeping v5 for Stretch is the right choice. I already uploaded a fix for the other good actionable report that Wookey made while he was looking at all this. > Are there any other regressions that you are aware of in v6? In terms of the upstream code, the issues with htags are the main 'showstopper' I'm currently aware of. But I haven't tested the rest of it anywhere near exhaustively yet either. In terms of the package currently in experimental, there's a bunch of issues with it that would need to be fixed if we were going to want it in Stretch. Wookey already identified some. And in the quick look that I've already had at it so far, there's others which were immediately obvious. There's a bunch of upstream stuff that is simply missing from it, the man page for htags-server, the icons and javascript and css for htags (which it actually complains loudly about when you try to use it, and makes me wonder how much they did actually test that or pay attention to what it reported) but that becomes moot if we go for the "remove htags completely" option too. There's little things like it still having .la files in it, and static libs for things that are supposedly 'plugins'. Which aren't a big deal on their own to fix, but again suggests that if 'obvious' things like that were missed, there's a good chance there will be more issues than what I've already seen in a cursory review. > Your suggestion, as I understand it, is that v6 should hit unstable > after stretch's release, and that people who are currently complaining > about bugs/missing-features in v5 (or are overly focused on numbers) can > then grab v6 out of stretch-backports. If we're going to say that our patience is up with waiting for sanity to return to htags (and mine pretty much is), then I think that's the most orderly way we can drain the swamp. With the least risk of having people step on the unexploded ordinance we'll expose, in ways that we can't easily fix under stable release constraints. I don't think it's the only option, but it looks like the least painful overall for everyone that I can so far think of, which maximises our options in the event of something unexpected (which ironically, are black swans we probably should expect ;) > Could you consider the relative merits of instead putting v6 into > stretch now, and dealing with the people that are currently clinging to > v5 by pointing out that: > > 0) there are now other alternatives to htags that might suit them better. I genuinely think that's actually the best advice we/I can give anyone, regardless of what our "for the short term" choice ends up being. At least unless somebody suddenly starts making major improvements to htags to do things that doxygen can't/doesn't already do on its own. But I'm at that awkward age, where I'm too old to imagine everyone will take my advice, but not so cranky to think that anyone who doesn't should have abuse and scorn heaped upon them before being nuked from orbit ... So I'm pretty sure someone will still want spacebar heating, whatever we might say. The difference is just how much warning we give them that we are taking it away, as to whether we can feel that we were polite and considerate enough about how we did that or not. > 1) htags-server lets them run the CGI for local access. > > 2) htags can generate static content, and thus safely provide general > access while avoiding the need for a CGI I think I already covered those above? (so I'll go back to trying to be brief and let you ask if there's something I didn't :) > 3) If there is anyone that cannot do either for some reason, they can > install global v5 from jessie, pin it to avoid upgrades, and report > the reasons why they had to do this to the BTS. That is another option. Though it seems a bit backward from what would seem like 'normal advice', and means they will fall out of having security support before Stretch is EOL. It's basically equivalent to saying "just maintain your own local build". I'm not sure I'd really want to *suggest* that to people, or get them in the habit of thinking that it's an ok thing to do when upgrading Debian - if it wasn't an option they figured out for themselves (and hopefully properly understood the consequences of). > Have I missed some vital aspect of this? Knowing what other people don't know is one of the hardest questions in the world :) So I hope you'll ask if there's something I didn't shed the sort of light you were looking for on. But you're asking good questions, and exploring the same sort of options that that I'd been mulling over, so I think we're at least in the same universe, even if we don't quite see it the same. Optimising for multiple variables is one of the other hardest things in the world, so I don't see it as a failure of anybody if we don't (ever) all completely agree on what would be 'best' here. I'll be happy if enough of us have at least thought about it hard enough to show their answer is based on the actual variables at play here. The center of gravity of those options should at least be a local maxima, if not the ideal solution. > Is there a compelling reason to favour the theoretical users that might > want to stick with v5 over the actual users that we've been hearing ask > for v6? That kind of depends on what we decide to do about htags when we move to v6 (now or later). That there are users of htags is definitely not theoretical, there was a new bug reported against it just yesterday - and unlike some of the more strident voices in this bug log, it's from an actual user of global, who I've never spoken to before, and wasn't astroturfed in as 'reinforcement'. If we tell him, "we fixed your bug, by removing htags", I'm pretty sure his first thought isn't going to be "thank you!". If we tell him, "we fixed your bug, and we removed the insecure CGI, but you can still generate static markup that you can't search" - I don't know what his answer will be, because he hasn't told us if he does use that or not. Similarly if we rush a 'blind' patch into Stretch to remove that and accidentally break something it would be hard to fix because there wasn't enough time spent on testing it all, we're somewhere on the spectrum of Suboptimal. > If the TC were to achieve consensus that v6 should be in stretch, will > it be sufficient for us to inform you of that in order to make it so? > > I gather from what you just wrote that it would be sufficient. Yeah, if we've got enough people willing to put their neck on the block to say "I think we're better off with v6-only in Stretch, rather than both via the magic of -backports, and here's why the people who might get burned by that, and the problems that can/will create, aren't important enough to exercise any patience to minimise them", then I'd be a fool not to jump at the chance to let them take the rap for whatever backlash might occur :) What I'd _like_ to see a good consensus on though, is a little more than that - because I don't think "should we ship v6 in Stretch" is the right question, or rather a _sufficient_ question. Because if the answer to that is "yes" - then we still have the question of "what should we do with htags in v6", and that's actually the one where things go all pear-shaped if we get it wrong. And even if the answer to it is "no", then that question _still_ remains as "what should we do with htags in v6 for stretch+1" ... So if we instead ask "what should we do with htags for Stretch (and beyond)", then everything else pretty much follows on automatically from that. If we say it's important to keep it as fully functional as it is now through the Stretch cycle, then really the only practical option is to stay with v5 for that. v6 has gutted too much of that to realistically fix in the time we have (or perhaps ever if upstream continues to fight that). If we say it's ok to lose the search function of it, or to just lose it altogether since the search was what made it useful - then we have nothing to gain by staying with v5 and should ship v6, and accept whatever fallout occurs from us doing that without any notice. Maybe that's what I somehow failed to communicate earlier when I kept saying htags was the important question I wanted to see get addressed. Because people saying "it's irrelevant, uploading 'something newer' is overwhelmingly more important" completely misses the point that uploading something newer unavoidably involves someone answering that question. And ignoring the issues around it totally leads to fun paradoxes like Odyx saying that one of the reasons it's important to have v6 is that it "fixes bugs like #590053" - when what the reporter of that bug wanted fixed is _exactly_ what we would lose by going to the htags from v6, and he was one of the people who also spent a considerable amount of time trying to work with upstream to have a secure system install of the CGI supported ... Or like Ian's faster than light "no brainer", where if we'd followed his advice, and made a snap decision without thinking to just let Vincent upload what Punit had prepared, because Clearly they were more capable than Evil Me of making an appropriate decision - then we'd have had a package in stretch with a trivial remote shell exploit in it ... Sam's argument that "we don't care" might be justifiable given the time that's elapsed and the direction upstream has trended in, but he hasn't yet given any preference for whether we remove htags entirely, or just try to amputate the CGI - which only gets us halfway to _what_ we should upload if we go with v6. I think Tollef and I are somewhere near to on the same page, in that we think it would suck to burn existing users on short notice, but agree we can't prioritise them forever, and are open about exactly how we proceed from here. I get the feeling that you are leaning toward preferring v6, but you're asking good questions, so I am interested in how you weigh all this up. But I think if we can get a sense of people's preference for what to do with htags, then we collapse the waveform of what logically follows from that - and if we don't have as many opinions as we have people, a reasonable consensus should be fairly quickly reachable from there, with a self-evident plan of action. And faster than it has while each person has focussed on their own personal subset of What Is Most Important here. > If however you are likely to insist on resolutions to override the > maintainer, or transfer the maintainership, I think you ought to be > up-front about that in order to avoid any accusations of intentional > time-wasting later on. I'm pretty sure that if wasting time was my intention, there'd be far more fun ways to do it than this! I know people here love their FAOD clarifications, so it helps you for us to be crystal clear about that, but it does make me sad that people do project those sort of accusations onto others. Especially onto others who *are* being diligent about making good decisions. I'm taking the time to give detailed answers to good questions because that's how good decisions get made. Anyone who thinks that 'threats' of "over my dead body" end with anything better than dead bodies, still has a lot to learn about how the world works. I'm pretty sure we can agree on some reasonable plan without the need for a vote, or to invoke the constitution to force anything. And I can guarantee you that on the remote chance that I'm wrong about that, and do violently disagree with some insane consensus that emerges - that I won't for a moment stand in the way of letting someone else be the person whose face that blows up in. If it comes to that, they'll have earned it! > If you can keep your answers brief, you'll earn my gratitude. Sorry :( I'll have to find some other way to earn that, if proving that your diligence in learning about this before deciding was well founded doesn't do it. Best, Ron
