Bug#935706: lintian: Make tag certainty a programmatic assessment
Hi Axel, > With removing the Certainty you basically removed the possibility to > write Lintian checks which are known to not be or in some cases even > never can be 100% perfect. I don't see how the removal of Certainty requires checks to be 100% perfect. There is no change of thought here, just the removal of metadata that was highly dubious to begin with. :) If a check is wildly inaccurate we still have many options available to us, including improving it (patches always welcome!), removing it, moving it to the "experimental" pile, or even adding an explicit remark to the tag's description, and so on. Indeed, this last idea will be much more useful than some implied subtle distinction between certainty levels. In practice, each particular new "certainty" value was not well- calibrated when added and essentially never adjusted … except in cases where the Lintian maintainers received bug reports that had an antagonistic quality to them fueled by an overly-optimistic appraisal of a check's reliability. Flames wrapped in the plausibly-deniable wrapper of a legitimate bug report aren't any more fun to receive than regular flames tbh. In other words, removal of the field simply reflects the reality and status quo that this field was misleading at best and inflammatory at worst. Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#935706: lintian: Make tag certainty a programmatic assessment
Hi Axel, [Thanks for the cc. I am on the list.] On Fri, Mar 20, 2020 at 4:57 PM Axel Beckert wrote: > > JFTR: I strongly disagree. The Certainty was a very helpful decision > helper for Lintian users who had a gut feeling about a lintian-emitted > tag being a false positive by seeing how reliable the author of the > tag suspected it's, well, Certainty — especially for heuristic checks. You are right. A lot of thought went into those ratings. There is a good likelihood we will revisit the issue. For example, we may implement an automatic tag reliability ranking based on the proportion of overrides in the archive. > > My impression is that the classification system is more fine-grained > > than the users of Lintian care about, At the same time, I agree even more strongly with Russ's sentiment. The system had too many levels. It was a burden to maintain. For each new tag, we had to look up what the result would be (i.e. error, warning, etc.). The new system is more direct. As one of the active Lintian maintainers, I don't think the concept of certainty is gone at all. First off, the fields are still around ( I am not sure about pkg-perl-tools). We just weakened the effect on each tag's everyday appearance—apparently based on a partial consensus, my apologies. In summary, I felt the concept of tag *certainty* fueled contentious (and unfortunately unfounded) debates over tag appearance. I am glad those days are gone. Going forward, the *reliability* of Lintian's heuristics, on the other hand, will become an important feature. It is an honest thing to do, and required for an expert system like Lintian. Kind regards Felix Lechner
Bug#935706: lintian: Make tag certainty a programmatic assessment
Hi again, taking Felix Lechner into Cc. Axel Beckert wrote: > to my dismay I discovered that Lintian's Certainty feature has been > removed. In addition to that I find it very bold to not even mention that removal explicitly in the debian/changelog entry: * Use the "Severity" field in tags to determine their display prominence directly. (Closes: #935706) I only found out about that removal via the bug report/fix against pkg-perl-tools. Regards, Axel -- ,''`. | Axel Beckert , https://people.debian.org/~abe/ : :' : | Debian Developer, ftp.ch.debian.org Admin `. `' | 4096R: 2517 B724 C5F6 CA99 5329 6E61 2FF9 CD59 6126 16B5 `-| 1024D: F067 EA27 26B9 C3FC 1486 202E C09E 1D89 9593 0EDE
Bug#935706: lintian: Make tag certainty a programmatic assessment
Hi,
to my dismay I discovered that Lintian's Certainty feature has been
removed.
Chris Lamb wrote:
> Controversial opinion
Indeed controversial.
> the "certainty" of tags is of no actionable benefit to either the
> users of Lintian or its developers and should be removed.
JFTR: I strongly disagree. The Certainty was a very helpful decision
helper for Lintian users who had a gut feeling about a lintian-emitted
tag being a false positive by seeing how reliable the author of the
tag suspected it's, well, Certainty — especially for heuristic checks.
With removing the Certainty you basically removed the possibility to
write Lintian checks which are known to not be or in some cases even
never can be 100% perfect.
One such check is for example library-package-name-for-application and
application-not-library which had the Certainty: wildguess as there's
no chance to be really precise. (Which probably is also the reason why
there are still tons of true positives out there, mostly python
packages.
> a) fixing the problem
> c) adding an override the issue
The Certainty field was a good help to decide between these two..
> the "certainty" is highly subjective and only appears
> to result in annoying our users when there is a legitimate false-
> positive and lintian is patronisingly and obstinately telling them it
> is "certain".
I disagree here, too. If it's a fixable false-positive, especially for
"Certainty: certain", then it should be fixed. If it's not fixable,
the Certainty should be downgraded.
Russ Allbery wrote:
> The problem, though, was that in some cases the bug would be a serious
> Policy violation *if Lintian were right*, but Lintian was often wrong.
> Certainty was an attempt to somehow capture that so that Lintian could
> express to the maintainer "this is a serious problem with your package if
> what I found is true, but there's a good chance this is a false positive."
This.
> From my perspective, the certainty concept is more useful for the
> wild-guess end of the spectrum, where it's conveying useful
> information ("this would be a serious problem, but we're bad at
> reliably detecting it").
This.
> In practice, I don't think this has happened.
It happened. These fields were used.
> My impression is that the classification system is more fine-grained
> than the users of Lintian care about,
Definitely not.
Regards, Axel
--
,''`. | Axel Beckert , https://people.debian.org/~abe/
: :' : | Debian Developer, ftp.ch.debian.org Admin
`. `' | 4096R: 2517 B724 C5F6 CA99 5329 6E61 2FF9 CD59 6126 16B5
`-| 1024D: F067 EA27 26B9 C3FC 1486 202E C09E 1D89 9593 0EDE
Bug#935706: lintian: Make tag certainty a programmatic assessment
Hi Felix, > > I would very much suggest we repurpose "severity" here instead of > > inventing a new term. > > That is exactly what I thought you might think (and it is why I > prepared a merge request instead of committing directly). > > Just one thought, please: Do you think the term severity may be, well, > too severe? Just to note up front that I fully understand your concern but I think we merely differ in degree in two key places. Firstly, I don't believe the term is — as you aptly (!) put it — all that severe to begin with. I would prefer to avoid tediously trading dictionary definitions with each other here but just to give one example from many, I would claim that "alert level" is a more emotionally-laden and critical term than "severity", but that may be an en_GB (or even an en_*) cultural assumption. Secondly, I think we place different values in consistency between older and newer versions of Lintian (ie. the cognitive cost of renaming this key term) as well as the benefits of harmonising with the rest of the Debian ecosystem in general. I tried I made most of my (admittedly somewhat rambling) point in my previous mail already so I will not bore readers with that again. :) Again, just to underline that I do understand the wider point you are trying to make about the philosophy of Lintian but I must stay with my original position that, on balance and after weighing both sides, retaining the term "severity" is the best way forward here. (Note from a practical point of view that this would not preclude us from renaming the term in the future if the situation changes; ie. this rename is somewhat orthogonal to the removal of the concept of "certainty".) Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#935706: lintian: Make tag certainty a programmatic assessment
[cc'ing the bug this time] Hi Chris, On Fri, Mar 13, 2020 at 4:58 PM Chris Lamb wrote: > > I would very much > suggest we repurpose "severity" here instead of inventing a new term. That is exactly what I thought you might think (and it is why I prepared a merge request instead of committing directly). Just one thought, please: Do you think the term severity may be, well, too severe? Ever since Michael Stapelberg called Lintian the policy's "programmatic embodiment" I have been trying to round its edges. (I even have a new logo idea, an L-shaped allen wrench instead of a square.) Is there not another, softer term like "alert level", "significance" or "relevance" we can use? Kind regards Felix On Fri, Mar 13, 2020 at 4:58 PM Chris Lamb wrote: > > Hi Felix, > > > > it appears to simply encode the currently unhelpful distinction between > > > "wild-guess", "possible" and "certain" in a new and relatively unfamiliar > > > way with a slightly ambiguous name. > > > > I think this is a case of miscommunication. > […] > > Any references to certainty or its values, "wild-guess", "possible" > > and "certain", are gone. The table you quote is being removed. > > Ah... very glad to hear we're on the same page here. I think I was > misled by the quoting of the table as it appeared to imply that > "certainty" or something similar would be retained, when that is not > what is being proposed here. > > Regarding the name of this new combined field, we should never forget > that is not only graphical applications that have a "user interface" — > even command-line utilities have one, but it is merely encoded in > ASCII form. Interface design is a long-established field and has > various hard-won best practices and conventions, more easily noticed > in GUI programs with regard to visual design, but an oft-neglected and > extremely important component of a good interface concerns itself with > the terminology used. For example, words should not surprise or > confuse the user, and indeed should be as entirely seamless and > unnoticable as possible. "Don't make me think", the saying goes. > Another way of putting this is that if the user even consciously has > to consider the word, it is ipso facto not a good word. > > I mention this because I believe "visibility" would not be serving our > users best. There are examples where a technically-correct name like > this, even if backed up by a well-meaning dictionary definition might > very well fit the idea better but we must have some empathy for the > casual and regular users of Lintian (a very old project, remember!) > who will be expecting the term "severity", even though that may not > 100% fit the idea of this new tag (eg. not quite matching the BTS or > whatever). In other words, if any user of Lintian now thinks questions > like "where did severities go? what is visibility? how does it > differ?" then we are not doing our job to the best of our ability. > > Anyway, unless I'm misunderstanding something again, I would very much > suggest we repurpose "severity" here instead of inventing a new term. > > > Regards, > > -- > ,''`. > : :' : Chris Lamb > `. `'` la...@debian.org chris-lamb.co.uk >`-
Bug#935706: lintian: Make tag certainty a programmatic assessment
Hi Felix, > > it appears to simply encode the currently unhelpful distinction between > > "wild-guess", "possible" and "certain" in a new and relatively unfamiliar > > way with a slightly ambiguous name. > > I think this is a case of miscommunication. […] > Any references to certainty or its values, "wild-guess", "possible" > and "certain", are gone. The table you quote is being removed. Ah... very glad to hear we're on the same page here. I think I was misled by the quoting of the table as it appeared to imply that "certainty" or something similar would be retained, when that is not what is being proposed here. Regarding the name of this new combined field, we should never forget that is not only graphical applications that have a "user interface" — even command-line utilities have one, but it is merely encoded in ASCII form. Interface design is a long-established field and has various hard-won best practices and conventions, more easily noticed in GUI programs with regard to visual design, but an oft-neglected and extremely important component of a good interface concerns itself with the terminology used. For example, words should not surprise or confuse the user, and indeed should be as entirely seamless and unnoticable as possible. "Don't make me think", the saying goes. Another way of putting this is that if the user even consciously has to consider the word, it is ipso facto not a good word. I mention this because I believe "visibility" would not be serving our users best. There are examples where a technically-correct name like this, even if backed up by a well-meaning dictionary definition might very well fit the idea better but we must have some empathy for the casual and regular users of Lintian (a very old project, remember!) who will be expecting the term "severity", even though that may not 100% fit the idea of this new tag (eg. not quite matching the BTS or whatever). In other words, if any user of Lintian now thinks questions like "where did severities go? what is visibility? how does it differ?" then we are not doing our job to the best of our ability. Anyway, unless I'm misunderstanding something again, I would very much suggest we repurpose "severity" here instead of inventing a new term. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#935706: lintian: Make tag certainty a programmatic assessment
Hi Chris, On Wed, Mar 11, 2020 at 10:58 AM Chris Lamb wrote: > > it appears to > simply encode the currently unhelpful distinction between "wild-guess", > "possible" and "certain" in a new and relatively unfamiliar way with > a slightly ambiguous name. I think this is a case of miscommunication. The new field, provisionally called Visibility, can have the values "error", "warning", "info", and "pedantic". They relate directly to the alert level, or phrased with less presumption, to the visibility of the diagnosis. Any references to certainty or its values, "wild-guess", "possible" and "certain", are gone. The table you quote is being removed. > As it still has a concept of how confident the Lintian maintainers > are, it will still be subject to the same "criticism" (as you call it) > and thus continue the unproductive and demotivating levels of > antagonism in otherwise legitimate bug reports that I am increasingly > seeing. Some old values for certainly are retained as text for research purposes. They have no functionality and will not show on the website. > My point all along is not that the concept of "certainty" has > zero value, but rather that it has a negative one. ie. it is a cost > and we should remove it. My merge request does exactly that. The concept of certainty is gone. There is no other way to do it. The merge request should be uncontroversial, except perhaps for the name of the new field. > As a minor point, I additionally would see no benefit in it being > customisable between/for teams and believe this would just be cause > for confusion when we attempt to reproduce & fix problems for our > users. This should really be discussed when it comes up, but for the record: the customization will relate to the alert level. Why, for example, should the golang team see 'package-has-long-file-name' about Joilet file names as a warning for their 'library' packages when those are unlikely to appear on a live CD? The method of detection is not customized. Everything will be reproducible unless suppressed, just like profiles work now. > Let us focus on making Lintian simpler & fun to maintain and > use instead. That seems like an odd thing to write. Is my commit https://salsa.debian.org/lintian/lintian/-/commits/92f2f155fb27abfb27523cab5df739bda3ebb552 not a step in that direction? Kind regards Felix Lechner
Bug#935706: lintian: Make tag certainty a programmatic assessment
Hi Chris,
On Sun, Mar 8, 2020 at 2:06 PM Chris Lamb wrote:
>
> As I understand it, you are proposing two interrelated changes here:
I can see why you might think that, but it is not how I would phrase it.
> a) Renaming the existing concept of severity (eg. error/warning/
> pedantic, etc.) to "visibility".
The field for severity does not hold those values. The alert levels
are calculated from this table:
# Map severity/certainty levels to tag codes.
our %CODES = (
classification => { 'wild-guess' => 'C', possible => 'C', certain => 'C' },
pedantic => { 'wild-guess' => 'P', possible => 'P', certain => 'P' },
wishlist => { 'wild-guess' => 'I', possible => 'I', certain => 'I' },
minor => { 'wild-guess' => 'I', possible => 'I', certain => 'W' },
normal=> { 'wild-guess' => 'I', possible => 'W', certain => 'W' },
important => { 'wild-guess' => 'W', possible => 'E', certain => 'E' },
serious => { 'wild-guess' => 'E', possible => 'E', certain => 'E' },
);
(from
https://salsa.debian.org/lintian/lintian/-/blob/master/lib/Lintian/Tag/Info.pm#L49-57)
I am suggesting to replace the fields for Severity and Certainty with
the values from that table (in word form). The proposed name for the
new field is Visibility, which is neutral and descriptive. (It also
offers no overtone of value; in fact, the visibility will become
completely customizable.)
> b) Dropping the certainty concept entirely (eg. wild guess, possible
> etc.)
Yes, the certainty will be dropped entirely, except I would like to
retain values other than 'certain' for a while to examine the accuracy
of the respective tags. Some thought (or criticism) went into those
values.
> Have I understood you correctly here?
Yes, with this clarification I am confident we are talking about the same thing.
Kind regards
Felix Lechner
Bug#935706: lintian: Make tag certainty a programmatic assessment
Felix Lechner wrote: > > Therefore I don't understand how introducing any new subjective tag, > > whatever its name, is the way forward here. > > The new field is not subjective. The alert level is not part of the > tag declarations. The new field, called Visibility, now determines the > alert level directly. It has a value of 'error', 'warning', 'info', > 'pedantic', or 'classification'. As I understand it, you are proposing two interrelated changes here: a) Renaming the existing concept of severity (eg. error/warning/ pedantic, etc.) to "visibility". b) Dropping the certainty concept entirely (eg. wild guess, possible etc.) Have I understood you correctly here? Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#935706: lintian: Make tag certainty a programmatic assessment
Hi Chris, On Fri, Mar 6, 2020 at 5:57 PM Chris Lamb wrote: > > Therefore I don't understand how introducing any new subjective tag, > whatever its name, is the way forward here. The new field is not subjective. The alert level is not part of the tag declarations. The new field, called Visibility, now determines the alert level directly. It has a value of 'error', 'warning', 'info', 'pedantic', or 'classification'. Here is a merge request that implements the proposal. Please let me know if you have any objections. https://salsa.debian.org/lintian/lintian/-/merge_requests/297 For details, please have a look at the commit messages. Kind regards Felix Lechner
Bug#935706: lintian: Make tag certainty a programmatic assessment
"Chris Lamb" writes:
> Controversial opinion — the "certainty" of tags is of no actionable
> benefit to either the users of Lintian or its developers and should be
> removed.
I may be able to provide a bit of historical context here, since I was
maintaining Lintian when this was introduced. This is not disagreement
with your proposal (I find your argument persuasive), just context so that
you know what problem we were trying to solve.
Prior to certainty, Lintian set the level of the tags directly. However,
they didn't have any well-defined meaning, and there was some confusion
and controversy over why a given tag would have a given level.
It seemed natural to tie the severity of the tag to the severity of the
bug that would be filed were one to file bugs based on lintian tags, since
that was tied into other project work and judgments (policy, BTS
conventions, and so forth). Debian maintainers already understood the
difference between serious, important, and normal, so reusing that
terminology seemed wise and it would put the level of each tag on more
concrete footing.
The problem, though, was that in some cases the bug would be a serious
Policy violation *if Lintian were right*, but Lintian was often wrong.
Certainty was an attempt to somehow capture that so that Lintian could
express to the maintainer "this is a serious problem with your package if
what I found is true, but there's a good chance this is a false positive."
The "certain" severity was always a problem; only a few things are truly
certain, since there are always special exceptions, and that's always
annoyed people. From my perspective, the certainty concept is more useful
for the wild-guess end of the spectrum, where it's conveying useful
information ("this would be a serious problem, but we're bad at reliably
detecting it").
A theory at the time was that maintainers could use these two metrics to
filter output on. Maintainers that only cared about actual Policy
violations, not Lintian's various advice, could filter on severity of
normal (or important), but allow any certainty. Maintainers who wanted
the advice but didn't want to be bothered with false positives could
enable any severity but filter out certainty of wild-guess and maybe even
possible.
In practice, I don't think this has happened. My impression is that the
classification system is more fine-grained than the users of Lintian care
about, so maintaining it is to some extent wasted effort.
--
Russ Allbery (r...@debian.org)
Bug#935706: lintian: Make tag certainty a programmatic assessment
Hi Felix, > > Controversial opinion — the "certainty" of tags is of no actionable > > benefit to either the users of Lintian or its developers and should > > be removed. > > Actually, I agree one hundred percent. I will introduce a new > field---probably called Relevance, Urgency, Significance, Importance, > Import, or simply Weight---to gradually replace the current mechanism. My concerns around the "certainty" field revolve around it being subjective and that, as outlined in my previous email, it makes ~zero difference to a maintainer's workflow. As a result, all discussions whether a particular "certainty" value is right or not is, at best, a distraction from Lintian's primary job. Therefore I don't understand how introducing any new subjective tag, whatever its name, is the way forward here. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#935706: lintian: Make tag certainty a programmatic assessment
Hi Chris, On Wed, Mar 4, 2020 at 6:00 PM Chris Lamb wrote: > > Controversial opinion — the "certainty" of tags is of no actionable > benefit to either the users of Lintian or its developers and should > be removed. Actually, I agree one hundred percent. I will introduce a new field---probably called Relevance, Urgency, Significance, Importance, Import, or simply Weight---to gradually replace the current mechanism. Please feel free to weigh in on the name. > Knowing the "certainty" is no meaningful benefit to working out what > to do from the above. Making the certainty programmatic is just > papering over this fundamental problem and implementation of this > feature could distract us from other, likely more valuable, tasks. Perhaps the certainty was introduced to deflect occasional but heated conversations about the diagnostic level. Furthering the same goal, I would like to propose another way to disentangle lab work from diagnosis. In my mind, Lintian's checks should primarily issue classification tags that state mundane facts. A later step would then issue a diagnosis based on the facts. That way, the initial scan is not subject any conversations other than method of detection. That line of thinking emerged when accommodating new tags for Debian Janitor. It is perhaps best illustrated by my commit: https://salsa.debian.org/lintian/lintian/-/commit/f4b977beb3ef777e5d1726512d077022ad348a8a Those tags simply list the fields present in debian/upstream/metadata. Based on discussions on IRC, such a basic tag made sense to Jelmer and offered greater utility in the long term. Sorry about the lack of communication. I noticed you added this commit later: https://salsa.debian.org/lintian/lintian/-/commit/00c29cf4b43c21edf1055cf9e06386e02e12e460 Did Jelmer ask for it? Debian Janitor would be a primary consumer of the simple lab work. Separating that from the actual diagnostics would permanently shift conversations about what should trigger a user alert to a higher level. User alerts could in fact be completely personalized, similar to the way profiles work today. People could trade profiles by email. We could even distribute snippets to some groups, such as Perl developers, who wish to receive higher alert levels for certain topics. One way to think about it, is that Lintian's lab work would be similar to a tokenizer like flex, and lintian's diagnostics would be more like bison. The exchange format would be JSON, so that it can be uploaded to UDD for everyone's benefit and potentially other tools, as well. > Even if there was some benefit or tortured situation where this might > affect what to do, the "certainty" is highly subjective and only appears > to result in annoying our users when there is a legitimate false- > positive and lintian is patronisingly and obstinately telling them it > is "certain". Thank you for bringing that up. It is perhaps the greatest point of all! Kind regards Felix Lechner
Bug#935706: lintian: Make tag certainty a programmatic assessment
Hi, > lintian: Make tag certainty a programmatic assessment Controversial opinion — the "certainty" of tags is of no actionable benefit to either the users of Lintian or its developers and should be removed. If a tag is emitted by Lintian, this leads to one of four actions: a) fixing the problem b) postponing the fixing of the issue c) adding an override the issue d) filing a false-positive bug report against lintian Knowing the "certainty" is no meaningful benefit to working out what to do from the above. Making the certainty programmatic is just papering over this fundamental problem and implementation of this feature could distract us from other, likely more valuable, tasks. Even if there was some benefit or tortured situation where this might affect what to do, the "certainty" is highly subjective and only appears to result in annoying our users when there is a legitimate false- positive and lintian is patronisingly and obstinately telling them it is "certain". Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#935706: lintian: Make tag certainty a programmatic assessment
Package: lintian
Hi,
A tag's certainty is currently fixed but actually depends on the
programmatic circumstances of its issuance. Some heuristics are better
than others. The 'tag' command should offer alternatives, which then
result in different alert levels (error, warning, info, and so on).
An example are tags that depend on native or non-native sources. For a
source package, the determination is easy. For an installation package
(*.deb) there is no way to know for sure, but the version number may
offer a guess:
For source packages, the tag would be certain (from a pending version
of checks/source-changelog.pm):
my $maintainer_revision = $latest_version->maintainer_revision;
(full version for native)
tag->certain 'hyphen-in-native-debian-changelog-version',
$maintainer_revision
if $info->native && $maintainer_revision =~ qr/-/;
while for installation packages (*.deb) it would not:
my $version = $latest_entry->{Version}; (literal version string)
tag->possible 'hyphen-in-native-debian-changelog-version', $version
if [not sure this can be determined];
The first use would produce an error; the second, a warning.
For the new mechanism to work, overrides should exclude the alert
level. They would function more like the universal tag format used in
the test suite.
Kind regards,
Felix Lechner
