[
https://issues.apache.org/jira/browse/GROOVY-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18077929#comment-18077929
]
Paul King edited comment on GROOVY-11983 at 5/3/26 10:19 PM:
-------------------------------------------------------------
What AI currently thinks is/isn't covered after this commit:
https://github.com/apache/groovy/commit/0f692ec17fe435049696cf9b43d36e00e4ab55c5
Summary:
Three categories of unsoundness this patch deliberately does *not* catch.
Captured for follow-up.
For context: STC tracks {{instanceof}} / {{!instanceof}} narrowing through a
stack of frames called the *temporary if-branch type information* (tti) — a
{{Map<Object, List<ClassNode>>}} held in
{{TypeCheckingContext.temporaryIfBranchTypeInformation}}. Each entry is either
*positive* (key is a {{Variable}} → "x is narrowed to T") or *negative* (key is
an {{Object[]}} starting with {{"!instanceof"}} → "x is known not to be T").
For the else branch of {{if (boolExpr) ... else ...}} (and the false branch of
a ternary, and the surrounding scope after an early-returning {{if (boolExpr)
return}}), {{visitIfElse}} / {{visitTernaryExpression}} *invert* the tti
captured during the visit of {{boolExpr}}: each positive entry becomes a
negative, and each negative is unwrapped back to a positive. The GROOVY-11983
fix gates that inversion behind a structural soundness check
({{canInvertNarrowingForElseBranch}}) on {{boolExpr}}.
A leading {{NotExpression}} ({{!E}}) is treated as unconditionally invertible:
{{visitNotExpression}} already sign-flips the operand's tti on the way out, the
else-branch inversion flips a second time, leaving entries that match {{E}}
being true — exactly the condition under which the else of {{if (!E)}} fires.
This rule is sound on its own, but it interacts with one of the categories
below.
h3. 1. Comparison/equality with an {{instanceof}} operand
{code:groovy}
if ((x instanceof Y) == flag) { ... } else { /* not invertible-soundly */ }
if ((x instanceof Y) != flag) { ... } else { ... }
{code}
{{!(A == B) <=> A != B}} — same XOR-style: knowing the equality is false
doesn't pin down either operand's truth value. The helper treats
{{==}}/{{!=}}/{{<}}/{{>}} etc. as leaves and returns {{true}}, so a
{{!instanceof}} hidden in such a comparison's operand could still be wrongly
unwrapped to a positive smart-cast (and emitted as a {{checkcast}} by static
compilation) in the else branch. Pathological — nobody writes {{(x instanceof
Y) == flag}} deliberately — so left as-is.
*Wrinkle from the {{NotExpression}} rule:* wrapping a
comparison-with-{{instanceof}} in a {{!}} (e.g. {{!((x instanceof Y) ==
flag)}}) elevates this category from "precision loss in the else" to a genuine
{{checkcast}}-style unsoundness. The negation route flips the positive entry
produced during the visit into a negative in the outer tti, which the else then
unwraps to a positive smart-cast. Still vanishingly rare in real code.
h3. 2. tti pollution through wrapper expressions the helper doesn't descend into
The helper only walks {{BinaryExpression}}, {{BooleanExpression}},
{{NotExpression}}, {{TernaryExpression}}. Anything else is a black-box leaf. So
an {{instanceof}} reachable only through:
* a method-call argument: {{if (foo(x instanceof Y && cond))}}
* a cast: {{if ((boolean)(x instanceof Y && cond))}}
* a closure call, range, etc.
... still pollutes the outer tti during the boolean visit (because
{{pushInstanceOfTypeInfo}} peeks the active tti frame regardless of where in
the AST it's called from), but the helper can't see the logical-AND inside and
returns {{true}}, so the unsound inversion fires. Fixing it properly would need
a generic AST walk; that has a much larger blast radius (false positives on
benign expressions) and the cases are vanishingly rare, so left as-is.
h3. 3. Coarse-grained verdict, not per-entry
When *any* AND-like node ({{&&}}, {{&}}, {{^}}) is found anywhere in the tree,
the helper returns {{false}} and the *entire* inversion for that else branch is
dropped. The {{NotExpression}} fix already closed the previous "{{!(A && B)}}
at top level wrongly disqualified" sub-case, but the disjunction shape is still
over-rejected:
{code:groovy}
if (z instanceof W || (cond && !(x instanceof Y))) { ... } else { ... }
{code}
The else implies {{!(z instanceof W) && !(cond && !(x instanceof Y))}} — so
{{z}} _can_ soundly be narrowed to "not W" in the else, even though the AND in
the right disjunct disqualifies the {{x}} entries. The current fix throws both
away. The principled fix is *per-entry tagging*: mark each tti addition with
whether an AND-like ancestor was on the path during visit, then only invert
untagged entries. That requires instrumenting {{visitBinaryExpression}} for
{{&&}}/{{&}}/{{^}} (push a "tainted" flag while visiting their operands) and
threading that flag through {{pushInstanceOfTypeInfo}} /
{{putNotInstanceOfTypeInfo}}. Not crash-causing — just precision loss in the
else branch.
h3. Summary
Crash-class bugs (the original GROOVY-11983 family — unsound {{checkcast}}
producing a runtime {{ClassCastException}}) are all handled. What remains are
(a) precision loss when an AND-like operator coexists with sound parts in a
disjunction, and (b) two pathological-shape unsoundness windows (comparison
with an {{instanceof}} operand, optionally elevated to crash-class when wrapped
in {{!}}; {{instanceof}} tunnelled through method-call args, casts, or other
non-boolean wrappers) that are theoretically real but practically not seen.
was (Author: paulk):
What AI currently thinks is/isn't covered (but it's pondering your
NotExpression comment right now):
Three categories of unsoundness this patch deliberately does *not* catch.
Captured for follow-up.
For context: STC tracks {{instanceof}} / {{!instanceof}} narrowing through a
stack of frames called the *temporary if-branch type information* (tti) — a
{{Map<Object, List<ClassNode>>}} held in
{{{}TypeCheckingContext.temporaryIfBranchTypeInformation{}}}. Each entry is
either *positive* (key is a {{Variable}} → "x is narrowed to T") or *negative*
(key is an {{Object[]}} starting with {{"!instanceof"}} → "x is known not to be
T"). For the else branch of {{if (boolExpr) ... else ...}} (and the false
branch of a ternary, and the surrounding scope after an early-returning {{{}if
(boolExpr) return{}}}), {{visitIfElse}} / {{visitTernaryExpression}} *invert*
the tti captured during the visit of {{{}boolExpr{}}}: each positive entry
becomes a negative, and each negative is unwrapped back to a positive. The
GROOVY-11983 fix gates that inversion behind a structural soundness check
({{{}canInvertNarrowingForElseBranch{}}}) on {{{}boolExpr{}}}.
h3. 1. Comparison/equality with an {{instanceof}} operand
{code:groovy}
if ((x instanceof Y) == flag) { ... } else { /* not invertible-soundly */ }
if ((x instanceof Y) != flag) { ... } else { ... }
{code}
{{!(A == B) <=> A != B}} — same XOR-style: knowing the equality is false
doesn't pin down either operand's truth value. The helper treats
{{{}=={}}}/{{{}!={}}}/{{{}<{}}}/{{{}>{}}} etc. as leaves and returns
{{{}true{}}}, so a {{!instanceof}} hidden in such a comparison's operand could
still be wrongly unwrapped to a positive smart-cast (and emitted as a
{{checkcast}} by static compilation) in the else branch. Pathological — nobody
writes {{(x instanceof Y) == flag}} deliberately — so left as-is.
h3. 2. tti pollution through wrapper expressions the helper doesn't descend into
The helper only walks {{{}BinaryExpression{}}}, {{{}BooleanExpression{}}},
{{{}NotExpression{}}}, {{{}TernaryExpression{}}}. Anything else is a black-box
leaf. So an {{instanceof}} reachable only through:
* a method-call argument: {{if (foo(x instanceof Y && cond))}}
* a cast: {{if ((boolean)(x instanceof Y && cond))}}
* a closure call, range, etc.
... still pollutes the outer tti during the boolean visit (because
{{pushInstanceOfTypeInfo}} peeks the active tti frame regardless of where in
the AST it's called from), but the helper can't see the logical-AND inside and
returns {{{}true{}}}, so the unsound inversion fires. Fixing it properly would
need a generic AST walk; that has a much larger blast radius (false positives
on benign expressions) and the cases are vanishingly rare, so left as-is.
h3. 3. Coarse-grained verdict, not per-entry
When *any* AND-like node ({{{}&&{}}}, {{{}&{}}}, {{{}^{}}}) is found anywhere
in the tree, the helper returns {{false}} and the *entire* inversion for that
else branch is dropped. That over-rejects:
{code:groovy}
if (z instanceof W || (cond && !(x instanceof Y))) { ... } else { ... }
{code}
The else implies {{!(z instanceof W) && !(cond && !(x instanceof Y))}} — so
{{z}} _can_ soundly be narrowed to "not W" in the else, even though the AND in
the right disjunct disqualifies the {{x}} entries. The current fix throws both
away. The principled fix is {*}per-entry tagging{*}: mark each tti addition
with whether an AND-like ancestor was on the path during visit, then only
invert untagged entries. That requires instrumenting {{visitBinaryExpression}}
for {{{}&&{}}}/{{{}&{}}}/{{{}^{}}} (push a "tainted" flag while visiting their
operands) and threading that flag through {{pushInstanceOfTypeInfo}} /
{{{}putNotInstanceOfTypeInfo{}}}. Not crash-causing — just precision loss in
the else branch.
h3. Summary
Crash-class bugs (the original GROOVY-11983 family — unsound {{checkcast}}
producing a runtime {{{}ClassCastException{}}}) are all handled. What remains
are (a) precision loss when an AND-like operator coexists with sound parts in
the same condition, and (b) two pathological-shape unsoundness windows
(comparison with an {{instanceof}} operand; {{instanceof}} tunnelled through
method-call args, casts, or other non-boolean wrappers) that are theoretically
real but practically not seen.
> STC: unsound smart-cast in else branch of if (cond && !(x instanceof Y))
> ------------------------------------------------------------------------
>
> Key: GROOVY-11983
> URL: https://issues.apache.org/jira/browse/GROOVY-11983
> Project: Groovy
> Issue Type: Bug
> Affects Versions: 5.0.5
> Reporter: Paul King
> Assignee: Paul King
> Priority: Major
>
> {{@CompileStatic}} unsoundly smart-casts a variable in the {{else}} branch
> when the {{if}} condition has the shape {{cond && !(x instanceof Y)}}. The
> else block is reached whenever {{cond}} is false (regardless of {{x}}'s
> type), but STC narrows {{x}} to {{Y}} and codegen emits {{checkcast Y}},
> throwing {{ClassCastException}} at runtime.
> h3. Reproducer
> {code:groovy}
> import groovy.transform.CompileStatic
> abstract class Animal { String name; boolean tame }
> class Cat extends Animal {}
> @CompileStatic
> class Tester {
> static String describe(Animal a) {
> if (a.tame && !(a instanceof Cat)) {
> return "tame non-cat $a.name"
> } else {
> return "other $a.name" // <-- a smart-cast to Cat here
> }
> }
> }
> Animal mystery = new Animal() {{ name = 'x'; tame = false }}
> Tester.describe(mystery) // ClassCastException: ... cannot be cast to Cat
> {code}
> {{javap}} of {{describe}} shows {{checkcast Cat}} on {{a}} in the else block
> before {{getName()}}.
> h3. Expected
> The else branch should not narrow {{a}} to {{Cat}}: it is reached when
> {{!a.tame || a instanceof Cat}}, so {{a}} is not guaranteed to be a {{Cat}}.
> h3. Workarounds
> * Replace {{!(x instanceof Y)}} with {{!Y.isAssignableFrom(x.getClass())}}
> * Split the conditional so the {{!instanceof}} stands alone
> h3. Root cause
> {{StaticTypeCheckingVisitor.visitIfElse}} inverts captured tti entries with
> {{tti.forEach(this::putNotInstanceOfTypeInfo)}}. The inversion is
> structurally blind: a {{!instanceof}} entry contributed by one operand of
> {{&&}} is unwrapped into the positive smart-cast bucket for the else branch,
> even though the else does not imply that operand was false. Sound for a
> single-operand boolean; unsound for compound conditions mixing {{&&}} with
> {{!instanceof}}.
> Related: GROOVY-6429, GROOVY-8412, GROOVY-8523, GROOVY-9931, GROOVY-11864
> (same {{tti}} plumbing); GROOVY-7971 fixed the symmetric then-branch issue
> for {{||}}.
> h3. Affects
> Groovy 5.0.x and 6.0.0-SNAPSHOT (master). Not present in 4.0.x.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
