Re: A pure Java example with reflective access behaving differently with invokeWithArguments() compared to core reflection's invoke() (Re: Strange observation: MethodHandle.invokeWithArguments() would not work, whereas Method.invoke() would with the very same arguments

[Peter Levart]

Hi Rony,

I showed you some code samples in Java language, so I had to explain 
what java compiler is doing too, for you to fully understand what's 
going on. You see, when using Java and HM.invokeWithArguments(), 
conversions (varargs included) are taking place at two levels. The 1st 
conversion is performed by javac generated bytecode (when invoking 
.invokeWithArguments, which is a varargs method) and the 2nd is the 
conversion performed by the HM.invokeWithArguments() method itself when 
it invokes the target method. The 1st is specified by JLS, while the 2nd 
is specified in the javadoc of .invokeWithArguments.

If we skip the 1st conversion, since you probably know what compiler is 
doing, you still have to be aware of the 2nd conversion, performed by 
MH.invokeWithArguments() (and MH.invoke()). First thing you have to know 
is the following:

/Method handles produced by lookups or constant loads from methods or 
constructors with the variable arity modifier bit (0x0080) have a 
corresponding variable arity, as if they were defined with the help of 
asVarargsCollector.//
/
This part does not mention the MethodHandles.Lookup.unreflect(Method) 
way of producing MethodHandle(s), but its all the same. You have to be 
aware that some method handles returned to you have a special property: 
they are varargs collectors. When invoking those methods, they accept 
any number of trailing positional arguments and collect them into an 
array argument.

This does not happen when you invoke such methods using reflection.

So if you want HM.invokeWithArguments() to behave the same as 
Method.invoke(), you have to transform the method handles obtained from 
lookup / unreflect with MH.asFixedArity() method. If the method handle 
is already a fixed arity method handle, this method will just return it, 
else it will return the equivalent method handle that doesn't perform 
collection of trailing positional arguments into an array argument. 
Using MH.asFixedArity() on each and every method handle you obtain by 
unreflect should give you the same behavior of MH.invokeWithArguments() 
as using Method.invoke().

Regards, Peter


On 06/10/18 14:16, Rony G. Flatscher wrote:

Dear Peter:

thank you very much for your thorough and extensive analysis taking 
the evolvement of Java into account!

You use the Java compiler and its behaviour (which adds syntax sugar 
to the Java language and is known to stick strictly to the Java 
language specification) for explanations and as reference, which is 
also what I have been doing in the past to counter-check the 
implemented behaviour of the bridge.

The Java compiler has a very different outset: *at compilation time* 
it knows everything it needs to know for compiling. If something is 
missing or wrong at compilation time it is able to stop compilation 
raising appropriate errors, thereby inhibiting the successful creation 
of Java classes as long as the Java programmer does not correct 
erroneous code or supplies missing vital information. Java being a 
strictly typed language allows one to cast arguments to force the 
compiler to pick the method the Java programmer had in mind in the 
case that the compiler could choose one from a set of a signatures.

By contrast a bridge for an interpreted, dynamically typed language 
(like ooRexx) needs to reflect and infer at *runtime* in order to be 
able to choose the Java methods with the appropriate signature. In 
this particular case all primitive datatypes are (ooRexx/C) strings in 
the dynamically typed language such that at runtime each argument 
needs to be checked whether they contain valid values for conversion 
to primitive datatypes in the signature of a Java method candidate.

In the dynamically typed language there is no means available for 
casting that would be needed to help solve the presented problem 
(there is optionally a box()-routine in the bridge that allows the 
ooRexx programmer to explicitly determine which primitive type is to 
be used for picking an appropriate Java method).

The new code in the bridge (had to rewrite that part from scratch 
because of Java 9 and later) uses java.lang.reflect to pick an 
appropriate Constructor, Method or Field object. For Java 1.6 and 1.7 
(although java.lang.invoke/MethodHandle got introduced with 1.7 I 
found cases where inaccessible types did not have accessible ancestors 
in the JRE) java.lang.reflect invocation will be carried out, for Java 
1.8 and later java.lang.invoke/MethodHandle invocations gets employed.

Comparing the speed between invocations using java.lang.reflect and 
java.lang.invoke, java.lang.invoke wins, but by a quite small margin.

Upon further testunit tests the rewritten bridge using 
java.lang.invoke would execute all ooRexx programs like 
java.lang.reflect since Java 1.1 with one single exception so far 
which I reported here, where invocation via java.lang.reflect behaves 
differently to java.lang.invoke.

Because of this discrepancy the question would be: is this considered 
a bug that will be corrected? Would there be otherwise a solution 
possible in the bridge that could take care of situations like this, 
where the bridge has no context-information other than referring to 
Constructor, Method or Field objects with the supplied arguments?

A few coarse ideas to address this:

  * if the unreflect() method gets used then MH.invoke() should behave
    like in the java.lang.reflect case; this means that the same
    assumptions should govern MH.invoke(): reason being that if a
    java.lang.reflect object gets unreflected, also the established
    java.lang.reflect rules should keep working in this case for the
    MH.invoke* in order to remain fully compatible with each other,
      o if a MH gets created without unreflect() then this would not
        be necessary
  * the bridge could restrain itself to only use java.lang.reflect for
    invocations on Java 8 and higher instead of java.lang.invoke
      o clearly, the current Java development efforts in this corner
        are concentrated on java.lang.invoke, so it would be desirable
        to switch from java.lang.reflect to java.lang.invoke which is
        not possible as long as this problem (different behaviour in
        j.l.r vs. j.l.i) persists
  * do not fix this, but document this as an incompatibility with the
    ramification that existing programs in dynamically typed languages
    need to be rewritten (e.g. in this case creating an Array object
    from the List and supply that instead)
  * ... ?

---rony

P.S.: It would be great if Java implemented reflective invocations for 
dynamically typed languages as this would solve the problem for all 
such languages once and forever in a standard manner, rather than have 
every bridge implementor create his own implementation, which is 
challenging,  effortful and possibly error prone. In my case the 
assumptions that governed the original implementation for reflective 
Constructor, Method and Field access for over fifteen years got broken 
by Java 9, because j.l.r (notable 'setAccessible') changed in a 
fundamental aspect. If Java could be emplyoed instead, the Java 
supplied implementation would be able to adopt changes in the inner 
workings accordingly thereby insulating bridges from such details and 
problems.


On 08.06.2018 18:23, Peter Levart wrote:
I think what you found is a discrepancy between handling of varargs 
methods using reflection vs. method handle's .invokeWithArguments().

Reflection basically ignores the fact that some methods are varargs 
methods. It treats them exactly the same as if they had an array 
parameter (not a varargs parameter).

For example, Arrays.asList() is a varargs method:

    public static <T> List<T> asList(T ... elements)

But reflection treats is exactly the same as the following method:

    public static <T> List<T> asList(T[] elements)

That's because varargs were introduced to Java in Java 5 as a kind of 
compilation sugar, implemented as an array parameter, while 
reflection is an older beast and wasn't updated to behave differently 
with varargs methods.

MethodHandle(s) came later, in Java 7, and took into consideration 
varargs methods.

So where's the problem? When you say in Java:

        Method asListMethod = Arrays.class.getMethod("asList", 
Object[].class);
        String[] elements = {"a", "b", "c"};

        System.out.println(
            asListMethod.invoke(null, elements)
        );

You get: IllegalArgumentException: wrong number of arguments

Because Method.invoke() is also a varargs method:

    public Object invoke(Object obj, Object... args)

... so java compiler (in order to be backwards source compatible with 
pre-varargs methods) treats this invocation in a way that just passes 
the String[] elements to the invoke method via the args parameter 
without any conversion (because String[] is a subtype of Object[]). 
Mathod.invoke then treats elements of the args[] array as individual 
verbatim parameters to be passed to the method when invoking it. As 
reflection treats Arrays.asList() as a normal no-varargs method which 
accepts a single Objet[] parameter and args[] contains 3 elements to 
be passed to a method with 3 parameters, exception occurs.


But when you use method handles and .invokeWithArguments():

        Method asListMethod = Arrays.class.getMethod("asList", 
Object[].class);
        MethodHandle asListMH = 
MethodHandles.lookup().unreflect(asListMethod);

        String[] elements = {"a", "b", "c"};

        System.out.println(
            asListMH.invokeWithArguments(elements)
        );

... the following happens: MethodHandle.invokeWithArguments() is also 
a varargs method:

    public Object invokeWithArguments(Object... arguments)

... so java compiler (in order to be backwards source compatible with 
pre-varargs methods) treats this invocation in a way that just passes 
the String[] elements to the invokeWithArguments method via the 
single 'arguments' parameter without any conversion (because String[] 
is a subtype of Object[]). MethodHandle.invokeWithArguments therefore 
takes 3 array elements and tries to invoke the underlying asList 
method with them. It observes that Arrays.asList is a varargs method, 
so it wraps the 3 "wannabe parameters" with the Object[] and passes 
the array to the asList method as single parameter. The result of 
above code is therefore:

    [a, b, c]

If you want MethodHandle.invokeWithArguments() to treat "wannabe 
parameters" passed to it as verbatim parameters regardless of the 
variable/fixed arity of the method to be invoked, then you can 
transform the variable arity MethodHandle to a fixed arity MH:

        Method asListMethod = Arrays.class.getMethod("asList", 
Object[].class);

        String[] elements = {"a", "b", "c"};

        System.out.println(
            asListMethod.invoke(null, (Object) elements)
        );

        MethodHandle asListMH = 
MethodHandles.lookup().unreflect(asListMethod);
        asListMH = asListMH.asFixedArity();

        System.out.println(
            asListMH.invokeWithArguments((Object) elements)
        );

Here you have it. Both of the above invocations produce equal output:

    [a, b, c]
    [a, b, c]


What happens here is the following (will only describe the MH case - 
Method case is similar):

MethodHandle.invokeWithArguments() is a varargs method:

    public Object invokeWithArguments(Object... arguments)

... so java compiler this time wraps the '(Object) elements' value 
with an Object[] because the value is of Object type (which is not a 
subtype of Object[]) - no backwards compatibility needed here, 
varargs conversion kicks in in the javac. 
MethodHandle.invokeWithArguments therefore takes an array with 1 
element (the element being 'elements' array) and tries to invoke the 
underlying asList method with it. This time it observes that the 
method is not a varargs method, (because MethodHandle was transformed 
to fixed arity method handle) so it passes the single 'elements' 
wannabee parameter to the Arrays.asList single Object[] parameter - 
this invocation works because fixed arity asList() takes Object[] and 
'elements' is a String[]...

Hope this helps you understand what's going on.

Regards, Peter

On 06/08/18 12:07, Rony G. Flatscher wrote:
On 11.03.2018 20:22, Rony G. Flatscher wrote:
Well, still trying to find out what the reason is, that core 
reflection's invoke behaves differently to MethodHandle's 
invokeWithArguments in one single case so far (using the method 
java.utli.Arrays.asList(...)).

Here is a little Java program that excercises reflective access to 
"java.util.Arrays.asList​(T... a)" using core reflection, i.e. 
"Method.invoke​(Object obj, Object... args)" and 
"MethodHandle.invokeWithArguments​(Object... arguments)":

    import java.util.*;

    import java.lang.reflect.*;
    import java.lang.invoke.*;

    class DemoAsListProblem
    {
         public static void main (String args[])
         {
             String arrNames[]=new String[] { "anne", "bert", "celine"};
             System.out.println("[1] (main) arrNames=\""+arrNames+"\", 
.toString()=\""+arrNames.toString()+"\"");
             List listNames=Arrays.asList(arrNames);
             System.out.println("[2] (main) after invoking Arrays.asList(arrNames), 
listNames=\""+listNames+"\"");
             System.out.println("\ninvoking testReflective() ...\n");

             testReflective();
         }

         public static void testReflective()
         {
             String arrNames[]=new String[] { "anne", "bert", "celine"};
             System.out.println("[3] (testReflective) arrNames=\""+arrNames+"\", 
.toString()=\""+arrNames.toString()+"\"");

             Method methAsList=null;
             List listNames=null;
             try {
                 Class paramTypes[]=new Class[] { (new Object[0]).getClass() };
                 methAsList=Arrays.class.getDeclaredMethod("asList", 
paramTypes);
                 System.out.println("--- (core reflection) Method object asList: 
"+methAsList);
                 System.out.println("\n--- (core reflection) now invoking 
Arrays.asList() via Method.invoke(...):");

                 listNames=(List) methAsList.invoke( null, (Object[]) new 
Object[]{arrNames} );   // static method
                 System.out.println("[4a] --- (CR) methAsList.invoke( null, (Object[]) new 
Object[]{arrNames} ) -> listNames=\""+listNames+"\"");

                 listNames=(List) methAsList.invoke( null, (Object)   new 
Object[]{arrNames} );   // static method
                 System.out.println("[4b] --- (CR) methAsList.invoke( null, (Object)   new 
Object[]{arrNames} ) -> listNames=\""+listNames+"\"");

    // "java.lang.IllegalArgumentException: wrong number of arguments":
    //            listNames=(List) methAsList.invoke( null, (Object[]) arrNames 
);   // static method
    //            System.out.println("[5a] --- (CR) methAsList.invoke( null, (Object[]) arrNames ) -> 
listNames=\""+listNames+"\"");

                 listNames=(List) methAsList.invoke( null, (Object)   arrNames 
);   // static method
                 System.out.println("[5b] --- (CR) methAsList.invoke( null, (Object)   arrNames )           
    -> listNames=\""+listNames+"\"");
             }
             catch (Throwable t)
             {
                 System.err.println("oops #1: "+t);
                 t.printStackTrace();
                 System.exit(-1);
             }

             System.out.println("\n--- (MH) now invoking Arrays.asList() via 
MethodHandle.invokeWithArguments(...):");
             MethodHandles.Lookup lookup=MethodHandles.lookup();
             MethodHandle mh=null;
             try {
                 mh=lookup.unreflect(methAsList);
                 System.out.println("--- (MH) unreflected MethodHandle mh: 
"+mh);

                 listNames=(List) mh.invokeWithArguments( (Object[]) new 
Object[]{arrNames} );
                 System.out.println("[6a] --- (MH) mh.invokeWithArguments(  (Object[]) new 
Object[]{arrNames} ) -> listNames=\""+listNames+"\"");

                 listNames=(List) mh.invokeWithArguments( (Object) new 
Object[]{arrNames} );
                 System.out.println("[6b] --- (MH) mh.invokeWithArguments(  (Object)   new 
Object[]{arrNames} ) -> listNames=\""+listNames+"\"");

                 listNames=(List) mh.invokeWithArguments( (Object[]) arrNames );
                 System.out.println("[7a] --- (MH) mh.invokeWithArguments(  (Object[]) arrNames )           
    -> listNames=\""+listNames+"\"");

                 listNames=(List) mh.invokeWithArguments( (Object) arrNames );
                 System.out.println("[7b] --- (MH) mh.invokeWithArguments(  (Object)   arrNames )           
    -> listNames=\""+listNames+"\"");

             }
             catch (Throwable t)
             {
                 System.err.println("oops #2: "+t);
                 t.printStackTrace();
                 System.exit(-2);
             }
         }
    }

Compiling and running it under 9.0.4 yields the following output:

    [1] (main) arrNames="[Ljava.lang.String;@27ddd392", 
.toString()="[Ljava.lang.String;@27ddd392"
    [2] (main) after invoking Arrays.asList(arrNames), listNames="[anne, bert, 
celine]"

    invoking testReflective() ...

    [3] (testReflective) arrNames="[Ljava.lang.String;@2a18f23c", 
.toString()="[Ljava.lang.String;@2a18f23c"
    --- (core reflection) Method object asList: public static java.util.List 
java.util.Arrays.asList(java.lang.Object[])

    --- (core reflection) now invoking Arrays.asList() via Method.invoke(...):
    [4a] --- (CR) methAsList.invoke( null, (Object[]) new Object[]{arrNames} ) -> 
listNames="[anne, bert, celine]"
    [4b] --- (CR) methAsList.invoke( null, (Object)   new Object[]{arrNames} ) -> 
listNames="[[Ljava.lang.String;@2a18f23c]"
    [5b] --- (CR) methAsList.invoke( null, (Object)   arrNames )               -> 
listNames="[anne, bert, celine]"

    --- (MH) now invoking Arrays.asList() via 
MethodHandle.invokeWithArguments(...):
    --- (MH) unreflected MethodHandle mh: MethodHandle(Object[])List
    [6a] --- (MH) mh.invokeWithArguments(  (Object[]) new Object[]{arrNames} ) -> 
listNames="[[Ljava.lang.String;@2a18f23c]"
    [6b] --- (MH) mh.invokeWithArguments(  (Object)   new Object[]{arrNames} ) -> 
listNames="[[Ljava.lang.Object;@13a57a3b]"
    [7a] --- (MH) mh.invokeWithArguments(  (Object[]) arrNames )               -> 
listNames="[anne, bert, celine]"
    [7b] --- (MH) mh.invokeWithArguments(  (Object)   arrNames )               -> 
listNames="[[Ljava.lang.String;@2a18f23c]"

So a String array is turned into a List using 
Arrays.asList(strArray). Doing it with core reflection yields 
different results to doing it with invokeWithArguments().

I would have expected that [4a] and [6a] would behave the same.

Using reflective invoke() and invokeWithArguments() has been 
working for my bridge for all the test units (using literally the 
same arguments in both cases) interchangeably, the one exception is 
Arrays.asList().

Maybe I am not seeing the obvious (having done too much "close-up 
tests" for quite some time now). So any hint, insight, help would 
be really appreciated!

---rony
As a few months have gone by without any follow-ups, I have been 
wondering whether there is a solution at all, if this is a problem 
rooted in the current implementation of MethodHandle methods not 
being 100% compatible with reflective invoke (which may imply that 
one needs to stick to use reflective invoke and forgo MethodHandle 
invokes for good).

---rony


_______________________________________________
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev