[
https://issues.apache.org/jira/browse/GROOVY-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18077929#comment-18077929
]
Paul King commented on GROOVY-11983:
------------------------------------
What AI thinks is/isn't covered:
Three categories of unsoundness this patch deliberately does *not* catch.
Captured for follow-up.
For context: STC tracks {{instanceof}} / {{!instanceof}} narrowing through a
stack of frames called the *temporary if-branch type information* (tti) — a
{{Map<Object, List<ClassNode>>}} held in
{{TypeCheckingContext.temporaryIfBranchTypeInformation}}. Each entry is either
*positive* (key is a {{Variable}} → "x is narrowed to T") or *negative* (key is
an {{Object[]}} starting with {{"!instanceof"}} → "x is known not to be T").
For the else branch of {{if (boolExpr) ... else ...}} (and the false branch of
a ternary, and the surrounding scope after an early-returning {{if (boolExpr)
return}}), {{visitIfElse}} / {{visitTernaryExpression}} *invert* the tti
captured during the visit of {{boolExpr}}: each positive entry becomes a
negative, and each negative is unwrapped back to a positive. The GROOVY-11983
fix gates that inversion behind a structural soundness check
({{canInvertNarrowingForElseBranch}}) on {{boolExpr}}.
h3. 1. Comparison/equality with an {{instanceof}} operand
{code:groovy}
if ((x instanceof Y) == flag) { ... } else { /* not invertible-soundly */ }
if ((x instanceof Y) != flag) { ... } else { ... }
{code}
{{!(A == B) <=> A != B}} — same XOR-style: knowing the equality is false
doesn't pin down either operand's truth value. The helper treats
{{==}}/{{!=}}/{{<}}/{{>}} etc. as leaves and returns {{true}}, so a
{{!instanceof}} hidden in such a comparison's operand could still be wrongly
unwrapped to a positive smart-cast (and emitted as a {{checkcast}} by static
compilation) in the else branch. Pathological — nobody writes {{(x instanceof
Y) == flag}} deliberately — so left as-is.
h3. 2. tti pollution through wrapper expressions the helper doesn't descend into
The helper only walks {{BinaryExpression}}, {{BooleanExpression}},
{{NotExpression}}, {{TernaryExpression}}. Anything else is a black-box leaf. So
an {{instanceof}} reachable only through:
* a method-call argument: {{if (foo(x instanceof Y && cond))}}
* a cast: {{if ((boolean)(x instanceof Y && cond))}}
* a closure call, range, etc.
... still pollutes the outer tti during the boolean visit (because
{{pushInstanceOfTypeInfo}} peeks the active tti frame regardless of where in
the AST it's called from), but the helper can't see the logical-AND inside and
returns {{true}}, so the unsound inversion fires. Fixing it properly would need
a generic AST walk; that has a much larger blast radius (false positives on
benign expressions) and the cases are vanishingly rare, so left as-is.
h3. 3. Coarse-grained verdict, not per-entry
When *any* AND-like node ({{&&}}, {{&}}, {{^}}) is found anywhere in the tree,
the helper returns {{false}} and the *entire* inversion for that else branch is
dropped. That over-rejects:
{code:groovy}
if (z instanceof W || (cond && !(x instanceof Y))) { ... } else { ... }
{code}
The else implies {{!(z instanceof W) && !(cond && !(x instanceof Y))}} — so
{{z}} _can_ soundly be narrowed to "not W" in the else, even though the AND in
the right disjunct disqualifies the {{x}} entries. The current fix throws both
away. The principled fix is *per-entry tagging*: mark each tti addition with
whether an AND-like ancestor was on the path during visit, then only invert
untagged entries. That requires instrumenting {{visitBinaryExpression}} for
{{&&}}/{{&}}/{{^}} (push a "tainted" flag while visiting their operands) and
threading that flag through {{pushInstanceOfTypeInfo}} /
{{putNotInstanceOfTypeInfo}}. Not crash-causing — just precision loss in the
else branch.
h3. Summary
Crash-class bugs (the original GROOVY-11983 family — unsound {{checkcast}}
producing a runtime {{ClassCastException}}) are all handled. What remains are
(a) precision loss when an AND-like operator coexists with sound parts in the
same condition, and (b) two pathological-shape unsoundness windows (comparison
with an {{instanceof}} operand; {{instanceof}} tunnelled through method-call
args, casts, or other non-boolean wrappers) that are theoretically real but
practically not seen.
> STC: unsound smart-cast in else branch of if (cond && !(x instanceof Y))
> ------------------------------------------------------------------------
>
> Key: GROOVY-11983
> URL: https://issues.apache.org/jira/browse/GROOVY-11983
> Project: Groovy
> Issue Type: Bug
> Affects Versions: 5.0.5
> Reporter: Paul King
> Assignee: Paul King
> Priority: Major
>
> {{@CompileStatic}} unsoundly smart-casts a variable in the {{else}} branch
> when the {{if}} condition has the shape {{cond && !(x instanceof Y)}}. The
> else block is reached whenever {{cond}} is false (regardless of {{x}}'s
> type), but STC narrows {{x}} to {{Y}} and codegen emits {{checkcast Y}},
> throwing {{ClassCastException}} at runtime.
> h3. Reproducer
> {code:groovy}
> import groovy.transform.CompileStatic
> abstract class Animal { String name; boolean tame }
> class Cat extends Animal {}
> @CompileStatic
> class Tester {
> static String describe(Animal a) {
> if (a.tame && !(a instanceof Cat)) {
> return "tame non-cat $a.name"
> } else {
> return "other $a.name" // <-- a smart-cast to Cat here
> }
> }
> }
> Animal mystery = new Animal() {{ name = 'x'; tame = false }}
> Tester.describe(mystery) // ClassCastException: ... cannot be cast to Cat
> {code}
> {{javap}} of {{describe}} shows {{checkcast Cat}} on {{a}} in the else block
> before {{getName()}}.
> h3. Expected
> The else branch should not narrow {{a}} to {{Cat}}: it is reached when
> {{!a.tame || a instanceof Cat}}, so {{a}} is not guaranteed to be a {{Cat}}.
> h3. Workarounds
> * Replace {{!(x instanceof Y)}} with {{!Y.isAssignableFrom(x.getClass())}}
> * Split the conditional so the {{!instanceof}} stands alone
> h3. Root cause
> {{StaticTypeCheckingVisitor.visitIfElse}} inverts captured tti entries with
> {{tti.forEach(this::putNotInstanceOfTypeInfo)}}. The inversion is
> structurally blind: a {{!instanceof}} entry contributed by one operand of
> {{&&}} is unwrapped into the positive smart-cast bucket for the else branch,
> even though the else does not imply that operand was false. Sound for a
> single-operand boolean; unsound for compound conditions mixing {{&&}} with
> {{!instanceof}}.
> Related: GROOVY-6429, GROOVY-8412, GROOVY-8523, GROOVY-9931, GROOVY-11864
> (same {{tti}} plumbing); GROOVY-7971 fixed the symmetric then-branch issue
> for {{||}}.
> h3. Affects
> Groovy 5.0.x and 6.0.0-SNAPSHOT (master). Not present in 4.0.x.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
