Re: Bug#841294: Overrule maintainer of "global" to package a new upstream version

[Wookey]

On 2016-11-30 16:56 +, Ian Jackson wrote:

> On to some questions it raises for me:
> 
> > Optionally, "htags" can generate a dynamic index (which reduces disk
> > space usage) but requires an http server setup to be able to serve the
> > pages. In this scenario, you will also need to be able to execute CGI
> > scripts as the symbol lookup is done when serving the http request (as
> > opposed to pre-processed when using static pages).
> > 
> > And this last bit (integration with system web server) is the
> > functionality that had security concerns raised by Ron [etc.]
> 
> So, to be clear, it is this functionality which is dropped in the
> package that you and Wookey uploaded to experimental/delayed ?

Said package is available as of today:
https://buildd.debian.org/status/package.php?p=global=experimental
 
And that functionality was left in as it was a self-contained patch,
pending determining whether in fact it remained useful/compatible or
not. So you still get htmake and htconfig commands.

However I have now done some checking, and no it doesn't work any more
because it uses the btreeop command, which has been removed in current
upstream, and the --action option to htags which is also gone.

We are having a think about whether this code can be hacked about a
bit to remain useful or if its approach is simply no longer relevant,
given upstream changes. And I just said replying to Sam, there is a
lot be said for not diverging significantly from upstream, because
there is significant overhead in doing so. So this patch is currently
likely to get dropped unless someone works out a way to make it
work/be useful.

> But AIUI this functionality remains:
> 
> > In addition to the above mechanisms, upstream also provides "htags"
> > which can be used to generate an HTML version of the index. "htags"
> > uses the index created by "gtags" and the source tree as input and
> > generates html files which allow you to navigate the source code in
> > the browser. By default, htags generates static pages which means you
> > can browse the code in a local browser without requiring a web server.

> So the impact for a user of the existing functionality the regression
> is not that their entire workflow is disrupted.
 
> Rather (unless the software is improved) they have these choices:
> 
>  - Put up with pregenerated html indices.  Is the only downside
>of this that they use additional disk space, and presumably
>cpu time to generate them ?

Correct. And they can easily be symlinked from say
/var/www/html/global/ to make them a) accessible from system
webserver and b) conveniently listed. The static HTML is 7G for a
kernel source tree. The dynamic version is 3.3G. (or 4.8G with the
suggested '--suggest2' options) So there is a x2 space overhead.

>  - Run the htags server on a high-numbered port somehow.  (Is there
>some kind of simple scriptery provided, to do this?)

There is an example in the NEWS file, which should go in the man page. It's 
trivial:

cd HTML; python -m CGIHTTPServer
or
cd HTML; python3 -m http.server --cgi

Then browse to http://localhost:8000/

>  - If the machine is not really multi-user, tear down the security
>boundary defending the webserver from their user account, and give
>their user account access to the webserver cgi directory (or plumb
>it in with symlinks).  (Are there any instructions or scripts for
>this?)  (Also this assumes that the source code is not super
>secret.)

I've just started playing with this to see what can be done (bearing
in mind the question of how much effort we should put into this sort
of upstream divergence anyway).

Simply symlinking /var/www/global/sourcetree to  
from 
gives you a convenient list of globalised htags to look through if they are 
static.

The issue with dynamic htags (and the search functionality) is that by
default your web server won't run .cgi scripts in arbitrary
directories (for good reasons). I don't know if we can have a central
.cgi that 'sources' the global.cgi or completion.cgi in the htags
sourcetree/HTML dir and uses it, or if that's still not actually a
good idea.

Possibly something nifty with apache config scripts would work. Suggestions 
welcome.

> I don't know much about this, but several of these choices seem likely
> to be plausible choices for many if not most current users of htags.

Right. I think the functionality upstream provides is fine.

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: Digital signature

Re: Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Wookey]

On 2016-11-30 11:39 -0500, Sam Hartman wrote:

> That is, I disagree with you that I need to address the question of
> htags and figure out whether htags users are being impacted.
> That might have been true for one release.
> But over a longer time frame, the really strong presumption is that we
> prefer a version that is being actively developed to one that isn't.
> That if people won't step forward and maintain htags, it goes awy.

Just to be clear, you are confusing 'htags' and 'htmake/htconfig' (I
was too when this started). htags is still here, works fine, and is
not going away, as Punit explained. The thing that is likely to go
away is 'centralised htags browsing using system webserver' which is
what Ron's htmake and htconfig provided.

> But I think what you're getting here from a lot of people is a belief
> that our community norm strongly favors active development and new
> software.  And sometimes when features are effectively not maintained in
> a manner that we can package them, they go away.

Right. We are a distro and by default we package what upstream
provides. A maintainer can choose to go further that that and
add/maintain extra functionality if they like, but it's an ongoing
workload that has to be carried. It is no doubt possible to modify
htmake/htconfig or write something equivalent to provide a centralised
index of 'globalified' source trees, but as upstream no longer
attempts to do such a thing there is a good argument that the debian
packaging shouldn't either. 

I certainly don't think that the fact that this was done once is a
good reason to stop packaging new releases.

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: Digital signature

Bug#841294: Overrule maintainer of "global" to package a new upstream version

[Punit Agrawal]

On Wed, Nov 30, 2016 at 4:56 PM, Ian Jackson
 wrote:
> Punit Agrawal writes ("Bug#841294: Overrule maintainer of "global" to package 
> a new upstream version"):
>> In offline discussions with Wookey, we came to the realisation that
>> reading the various bug reports (including this one) it is not very
>> clear how global (upstream) is structured, the functionality it
>> provides and bits that are holding back the debian update.
>
> Thank you for your clear and excellent explanation.
>
>
> On to some questions it raises for me:
>
>> Optionally, "htags" can generate a dynamic index (which reduces disk
>> space usage) but requires an http server setup to be able to serve the
>> pages. In this scenario, you will also need to be able to execute CGI
>> scripts as the symbol lookup is done when serving the http request (as
>> opposed to pre-processed when using static pages).
>>
>> And this last bit (integration with system web server) is the
>> functionality that had security concerns raised by Ron [etc.]
>
> So, to be clear, it is this functionality which is dropped in the
> package that you and Wookey uploaded to experimental/delayed ?
>

The debian packaging added two tools not present in upstream -
htconfig and htmake. These tools and debian patched htags together
make the integration with system web server work to the extent that
Ron was comfortable shipping the package. Both htmake and htconfig are
authored by Ron.

So although the package uploaded by Wookey retain the tools they have
become non-functional due to upstream changes to htags. But as you
point out, you can still achieve a very similar end result using other
mechanisms.

> But AIUI this functionality remains:
>
>> In addition to the above mechanisms, upstream also provides "htags"
>> which can be used to generate an HTML version of the index. "htags"
>> uses the index created by "gtags" and the source tree as input and
>> generates html files which allow you to navigate the source code in
>> the browser. By default, htags generates static pages which means you
>> can browse the code in a local browser without requiring a web server.
>
>
>
> So the impact for a user of the existing functionality the regression
> is not that their entire workflow is disrupted.

That is exactly what I was trying to get across in my previous email.

>
> Rather (unless the software is improved) they have these choices:
>
>  - Put up with pregenerated html indices.  Is the only downside
>of this that they use additional disk space, and presumably
>cpu time to generate them ?

There

>
>  - Run the htags server on a high-numbered port somehow.  (Is there
>some kind of simple scriptery provided, to do this?)

It would be a web server - htags isn't a server in itself. Upstream's
suggestion of using

python -m CGIHTTPServer

in the generated HTML folder worked for the package Wookey has
uploaded. This command can be executed with user privileges and runs a
local instance of an http server.

IIUC, running the web server with user privileges, prevents it from
binding to external interfaces/IP addresses.

>
>  - If the machine is not really multi-user, tear down the security
>boundary defending the webserver from their user account, and give
>their user account access to the webserver cgi directory (or plumb
>it in with symlinks).  (Are there any instructions or scripts for
>this?)  (Also this assumes that the source code is not super
>secret.)

So I am not very familiar with cgi scripts (having never used it
myself) but I believe what you say is possible - somebody with a
little more knowledge than me should be easily able to come up with
the instructions.

>
> I don't know much about this, but several of these choices seem likely
> to be plausible choices for many if not most current users of htags.
>
>
> FAOD none of this changes my view about the proper resolution of this
> TC petition.  But answers might help clarify the discussion.
>
> Thanks,
> Ian.
>
> --
> Ian Jackson    These opinions are my own.
>
> If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
> a private address which bypasses my fierce spamfilter.

Bug#841294: Global Ballot Thoughts

[Sam Hartman]

I'd really like to see the TC offer at least the following advice:

1) We believe that strong evidence is required to hold back integrating
new versions of software like global.  The burden of proof is on those
who propose not to update, not on those who would like Debian to contain
current upstream features.

2) This burden has not been met with regard to htags and regressions
related to htags.

3) Delays in discussion of this issue over the year suggest that having
more people involved in maintaining the global package would help
address  a perception that the maintainer is blocking forward progress.

I don't think I'd support giving global to someone else.  I don't think
we even need to say "Ron you did something bad."  I do think that Ron
contributed to  a harmful perception that damages those who would use and
contribute to global in Debian.
I think taking steps like involning others in the process would be good
in fighting that perception and would be good for the package at a
technical level.
If we can find a path forward that gets a new global into Debian, I'd be
happy only offering advice.  If we get stuck doing that, I think we need
to overrule something.

Bug#841294: Overrule maintainer of "global" to package a new upstream version

[Ian Jackson]

Punit Agrawal writes ("Bug#841294: Overrule maintainer of "global" to package a 
new upstream version"):
> In offline discussions with Wookey, we came to the realisation that
> reading the various bug reports (including this one) it is not very
> clear how global (upstream) is structured, the functionality it
> provides and bits that are holding back the debian update.

Thank you for your clear and excellent explanation.


On to some questions it raises for me:

> Optionally, "htags" can generate a dynamic index (which reduces disk
> space usage) but requires an http server setup to be able to serve the
> pages. In this scenario, you will also need to be able to execute CGI
> scripts as the symbol lookup is done when serving the http request (as
> opposed to pre-processed when using static pages).
> 
> And this last bit (integration with system web server) is the
> functionality that had security concerns raised by Ron [etc.]

So, to be clear, it is this functionality which is dropped in the
package that you and Wookey uploaded to experimental/delayed ?

But AIUI this functionality remains:

> In addition to the above mechanisms, upstream also provides "htags"
> which can be used to generate an HTML version of the index. "htags"
> uses the index created by "gtags" and the source tree as input and
> generates html files which allow you to navigate the source code in
> the browser. By default, htags generates static pages which means you
> can browse the code in a local browser without requiring a web server.



So the impact for a user of the existing functionality the regression
is not that their entire workflow is disrupted.

Rather (unless the software is improved) they have these choices:

 - Put up with pregenerated html indices.  Is the only downside
   of this that they use additional disk space, and presumably
   cpu time to generate them ?

 - Run the htags server on a high-numbered port somehow.  (Is there
   some kind of simple scriptery provided, to do this?)

 - If the machine is not really multi-user, tear down the security
   boundary defending the webserver from their user account, and give
   their user account access to the webserver cgi directory (or plumb
   it in with symlinks).  (Are there any instructions or scripts for
   this?)  (Also this assumes that the source code is not super
   secret.)

I don't know much about this, but several of these choices seem likely
to be plausible choices for many if not most current users of htags.


FAOD none of this changes my view about the proper resolution of this
TC petition.  But answers might help clarify the discussion.

Thanks,
Ian.

-- 
Ian Jackson    These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Sam Hartman]

> "Ron" == Ron   writes:

Ron> Hi OdyX,

Ron> On Wed, Nov 16, 2016 at 03:23:47PM +0100, Didier 'OdyX' Raboud wrote:
>> Hi there,
>> 
>> I've been mostly VAC, and only now found enough time to properly
>> read through this bug log. In the interest of transparency and to
>> help the TC reach a consensus (also through understanding what
>> the other members' understanding, ideas and positions are), here
>> comes my current understanding of the problem at hand.
>> 
>> As preamble, I will upfront state that I have _not_ looked in the
>> precise details of the so-called 'htags' functionality, as, the
>> rest will show, my current viewpoint on the problem is that it
>> doesn't matter.

Ron> If we run with your proposal, what are you actually suggesting
Ron> we tell the people who'd be upset by the loss of htags without
Ron> notice in Stretch?  Because I don't really see how you've
Ron> addressed that here.

Ron, thanks for working with the process.

I think that one thing you sometimes find when you try and build
consensus is that your conception of the problem and even the values by
which a solution should be judged is not shared.

As I understand it, you believe we should be evaluating the technical
options and that preserving things that work is of high value.

I think a lot of us believe that get newest code  is a presumptive value
especially over the multiple releases time frame.

That is, I disagree with you that I need to address the question of
htags and figure out whether htags users are being impacted.
That might have been true for one release.
But over a longer time frame, the really strong presumption is that we
prefer a version that is being actively developed to one that isn't.
That if people won't step forward and maintain htags, it goes awy.

That's a presumption.  If someone gets evidence that htags is heavily
used, we can consider that.
We might even go so far given sufficient evidence to decide that forking
global and never taking a new upstream was the right answer for our
users.  That would take a lot of evidence.

I disagree with your approach that the people wanting to remove htags
need to show that it is being used.  I disagree with your view that over
the timescales involved, people wanting a new upstream need to justify
that.

Yes, removing htags creates a regression.

Yes, I'm effectively saying "If you're using htags, sucks to be you.  If
you're willing to spend effort maintaining a version of htags that is
secure, then we might be able to bring it back.

Yes, it's possible that we'll learn we broke things and need to revert.

But I think what you're getting here from a lot of people is a belief
that our community norm strongly favors active development and new
software.  And sometimes when features are effectively not maintained in
a manner that we can package them, they go away.

I don't think it's reasonable to defer this to upstream and wait until
upstream removes htags.

I have tried to understand the technical issues.  I'm not sure I'm 100%
in agreement in your analysis there, but I'm fairly close.  However, I
haven't found the technical issues tend to affect my analysis of what
Debian should do.


I'd strongly urge you to work on a global6 package.  I don't care
whether it's called global or global6.  I don't think it should include
insecure cgi scripts that are enabled by default.  I'd be fine if it
didn't include htags at all.
I'm not saying upload that package now; I'm not saying where to upload
it.
(Although I wouldn't object to an upload to experimental or unstable)
I think having that package ready and keeping options open as long as
possible would help preserve our ability to work through this process.
I hope that would go a long way to addressing people's feelings of
frustration.

Thanks for your consideration,

--Sam

Bug#841294: Overrule maintainer of "global" to package a new upstream version

[Punit Agrawal]

On Wed, Nov 23, 2016 at 5:19 PM, Didier 'OdyX' Raboud  wrote:
> The TC had an IRC meeting yesterday [0], and I (was) volunteered to wrap up
> the different outcome possibilities, which would help forming our opinions.
> Not all these options are exclusive, or would need an actual TC decision.
>
> Here we go:
>
> A) 'global' stays maintained as it currently is.
>
> This would imply:
>  - no new 'global' upstream release before stretch,
>  - no 'htags removal' warning in stretch,
>  - possibility for the maintainer (and/or other interested parties) to start
>maintaining a 'global6'; this wouldn't offer any guarantee for a more
>recent version of 'global' in stretch. This would also imply having two
>'global' packages in certain suites.
>
> B) A fresher version of 'global' is uploaded to experimental 'soon'
>(with or without interested parties' help; with or without a TC decision)
>
> This would imply:
>  - any interested party would file (and close) Debian bugs for issues and
>regressions (with appropriate severities), to make the 'fitness for a
>stable release' assessment easier, and earlier.
>
> C) After the release of stretch, a fresher version of 'global' is uploaded to
>unstable with the explicit goal of making it available in buster.
>
> This would imply:
>  - any interested party would file (and close) Debian bugs for issues and
>regressions (with appropriate severities), to make the 'fitness for a
>stable release' assessment easier.
>  - after migration to testing, this would make the fresher version of 'global'
>available for backporting to stretch-backports.
>  - the version of 'global' released in stretch could carry 'htags removal'
>warnings;
>
> D) A fresher version of 'global' is uploaded to unstable 'soon', targetting
>stretch (with or without interested parties' help; with or without a TC
>decision)
>
> This would imply:
>  - overruling the 'global' maintainer's decision (§6.1.4, implies 3:1 majority
>in a TC vote);
>  - any interested party (including the maintainer) would file (and close)
>Debian bugs for issues and regressions (with appropriate severities), to
>make the 'fitness for a stable release' assessment easier.
>  - that this fresher version of 'global' would reach 'fit for release' status
>before the Stretch release.
>
> E) the 'global' package is handed to other maintainer(s)
>
> This would imply:
>  - overruling the 'global' maintainer's decision (§6.1.4, implies 3:1 majority
>in the TC vote);

In offline discussions with Wookey, we came to the realisation that
reading the various bug reports (including this one) it is not very
clear how global (upstream) is structured, the functionality it
provides and bits that are holding back the debian update.

Gnu Global is a source code tagging system similar in functionality to
ctags or cscope. It allows searching for symbol definitions and use in
the codebase by splitting the functionality into two steps -

* Indexing - This is a offline step where an index of source code
symbols definitions and references for a code base is created. This
functionality is provided by "gtags" binary (part of upstream global).
Indexes (or is it indices?) typically live in the source tree (they
don't need to).

* Searching - Once the code base is indexed, the indexes can be used
to search for symbol definitions as well as query locations where the
symbol is referenced. This can be done using the "global" binary
(again provided by upstream). Also there are various integrations with
commonly used editors - emacs, vim, elvis, etc - that internally
invoke global for their symbol lookup requirements. So if you are a
developer who would like to use global to navigate your way through a
large code base a common way to use it would be to use "gtags" and the
editor integration of your choice.

In addition to the above mechanisms, upstream also provides "htags"
which can be used to generate an HTML version of the index. "htags"
uses the index created by "gtags" and the source tree as input and
generates html files which allow you to navigate the source code in
the browser. By default, htags generates static pages which means you
can browse the code in a local browser without requiring a web server.

Optionally, "htags" can generate a dynamic index (which reduces disk
space usage) but requires an http server setup to be able to serve the
pages. In this scenario, you will also need to be able to execute CGI
scripts as the symbol lookup is done when serving the http request (as
opposed to pre-processed when using static pages).

And this last bit (integration with system web server) is the
functionality that had security concerns raised by Ron and for which
there are patches in the debian package. In recent versions (> 6.4,
Apr '15), upstream has dropped support for generating scripts that
expect to be written to system cgi-bin folder. As demonstrated by
wookey[1], it is still