Hi all,
A follow-up to my previous mail, since I have been thinking about this
some more and also found some relevant older discussions.
I think the `asType` example is actually exposing a more general problem
than just `asType`.
My current understanding is that if we say that an extension method is
conceptually similar to an instance method, then ideally it should
participate in dispatch in roughly the same way.
For example, suppose:
```
class A {
}
class B extends A {
def foo() {
"instance"
}
}
```
and there is an extension method:
```
foo(A)
```
Then for:
```
A x = new B()
x.foo()
```
I would expect the instance method on B to win, even though the static
type of x is A.
Likewise, if:
```
class C extends B {
}
```
and there is an extension method:
```
foo(C)
```
then for a runtime C I would expect the more specific C extension to win
over the A extension.
This is not usually a practical problem because most extension methods
are on interfaces or base classes which are not subsequently extended
with an instance method of the same signature. But it is still the
semantics I would expect if extension methods are supposed to behave
like methods added to the receiver.
There is an important qualification here, though: I don't think the
answer can simply be "use normal Groovy dynamic dispatch".
For `@CompileStatic`, I think it is generally desirable that arbitrary
MetaClass changes do not influence the compiler's decisions. That is one
of the reasons static compilation exists in the first place. Categories
and other runtime metaprogramming mechanisms raise similar questions.
So I am not suggesting that we should turn every extension-method call
under static compilation into a normal Groovy dynamic call site.
The problem is that, once we exclude the full dynamic MOP semantics, I
am not sure what the correct replacement semantics are.
For ordinary instance methods, the JVM already gives us the required
mechanism:
```
invokevirtual
```
The compiler can statically select A.foo as the symbolic target, while
the runtime receiver still determines which override is executed.
For an extension method, however, we have an `invokestatic`. Once we
have selected:
```
DGM.foo(A, x)
```
we have effectively already decided that this is the extension method to
call.
We can make extension-method selection smarter and, for example,
generate a dispatch table based on the runtime type of x. But that still
only dispatches between extension methods.
It cannot recover the possibility that x has an instance method which
should have taken precedence.
And if extension methods can come from multiple extension modules, there
is another problem: what is the complete set of extension methods which
participates in this dispatch in the first place?
This made me look back at some of the earlier discussions.
**GROOVY-8788** was already about making extension-method selection
under `@CompileStatic` agree with the dynamic runtime. Eric's change
made the static compiler prefer the same extension method as the dynamic
runtime, and the change was noted as potentially breaking:
https://issues.apache.org/jira/browse/GROOVY-8788
The discussion also makes the more general principle explicit: the
static compiler should behave consistently with dynamic Groovy in terms
of what extension methods mean.
But I think GROOVY-8788 only solves the extension-vs-extension part of
the problem. It does not solve instance-method-vs-extension-method
dispatch where the runtime receiver is a subtype of the static type.
I also realized that I had already started down a related path in
January in the `asBoolean` discussion:
https://www.mail-archive.com/dev%40groovy.apache.org/msg07596.html
There the question was whether Groovy Truth should effectively always
mean `foo.asBoolean()`, rather than having the compiler replace that
operation with various static/primitive/null fast paths.
That discussion also brought up using `invokedynamic` for the cases
where runtime dispatch is actually needed.
Looking at `asType` now, I think the issue is broader: even without
MetaClass changes, choosing the extension method statically can already
lose information which is relevant to the eventual dispatch.
There is another interesting comparison with the recent `*Assign` work /
GEP-15:
https://www.mail-archive.com/commits%40groovy.apache.org/msg27325.html
There the language-level operation is explicitly allowed to resolve to
an instance method, extension method or category method. Under static
compilation the proposal uses `findMethod()` and emits a direct call,
whereas dynamic Groovy uses the MOP.
That seems reasonable if static compilation is allowed to make the
operation static. But it also illustrates the question I am now running
into:
**when is `findMethod()` actually sufficient to represent the Groovy
semantics, and when does the operation inherently require some form of
runtime dispatch?**
For me this leaves roughly these possibilities:
1. We deliberately define extension methods under static compilation as
statically selected methods. This is simple and fast, but then extension
methods are not really virtual-like in the way I would intuitively expect.
2. We define a restricted virtual-like dispatch model for extension
methods. The runtime receiver can then influence the result, but only
within a precisely defined set of candidates. This would avoid full MOP
semantics, but we would need to define the candidate set and precedence
very carefully.
3. We emit a specialized dynamic call site for cases where the runtime
type can affect the result. This would still be dynamic semantically,
but could potentially be optimized/cached using `invokedynamic`. The
question would then be exactly which parts of Groovy's dynamic semantics
such a call site is supposed to implement.
4. We accept that `@CompileStatic` has a different semantic contract
here and document the differences.
The problem I see with (2) is that extension modules make this an
open-world problem.
If module M1 provides:
```
foo(A)
```
and module M2 provides:
```
foo(B)
```
then a compiler which only knows about M1 cannot necessarily construct
the complete dispatch hierarchy.
If we also allow an instance method on B to participate, we effectively
have to combine:
```
runtime class hierarchy
+ instance methods
+ extension methods
+ extension modules
```
into one dispatch decision, while deliberately *not* necessarily
including the full MetaClass/category machinery.
At that point I don't see how an ordinary `invokestatic` can preserve
the semantics.
On the other hand, I also don't think "just make it dynamic" is a
satisfactory answer for `@CompileStatic`, because arbitrary runtime
MetaClass changes should not normally be able to invalidate the static
compilation model.
So perhaps the interesting question is whether there is a well-defined
middle ground: a **static-compilation-specific dispatch model for
extension methods**, which has enough runtime information to preserve
virtual-like extension semantics but is still closed and deterministic
with respect to the parts of Groovy metaprogramming that
`@CompileStatic` is supposed to ignore.
I am particularly interested in what people think the intended contract
actually is here.
In particular:
* Is the intended model for extension methods under `@CompileStatic`
fundamentally "static overload resolution", or should they be considered
virtual-like methods for purposes of dispatch?
* If they are virtual-like, what should the precedence be between an
instance method and an extension method when the instance method only
exists on a runtime subtype?
* What should happen when the more specific extension method comes from
a different extension module?
* Is there already an established design decision around this which I am
missing?
* And, specifically for `asType` / `asBoolean`, should we regard these
as special language operations which are allowed to use a restricted
form of dynamic dispatch under static compilation?
There may well be historical context that is no longer obvious from the
current code or Jira issues.
My current feeling is that `asType` is only the symptom. The underlying
question is what semantics we actually want extension methods to have
under static compilation, and whether those semantics can be represented
by static method selection at all.
bye
Jochen
