Re: [perl #57190] HLL Interoperation

[Allison Randal]

Bob Rogers wrote:

   OK, here's my straw-man proposal for a language interoperability
framework; my apologies for sitting on it so long.  It's still pretty
messy, but I'm sure it will benefit more from other viewpoints at this
stage than from polishing.


Thanks, added to trunk as pdd31_hll_interop.pod in r30620. I gave it a 
quick review, but haven't edited it yet.


Allison

Re: [perl #57190] HLL Interoperation

[Allison Randal]

Allison Randal wrote:

Bob Rogers wrote:

   OK, here's my straw-man proposal for a language interoperability
framework; my apologies for sitting on it so long.  It's still pretty
messy, but I'm sure it will benefit more from other viewpoints at this
stage than from polishing.


Thanks, added to trunk as pdd31_hll_interop.pod in r30620. I gave it a 
quick review, but haven't edited it yet.


I've added a series of inline comments to help guide further discussion. 
Thanks for raising key points to get the discussion started.


Allison

Re: [perl #57190] HLL Interoperation

[Bob Rogers]

   OK, here's my straw-man proposal for a language interoperability
framework; my apologies for sitting on it so long.  It's still pretty
messy, but I'm sure it will benefit more from other viewpoints at this
stage than from polishing.

-- Bob Rogers
   http://rgrjr.dyndns.org/

# Copyright (C) 2008, The Perl Foundation.
# $Id: $

=head1 NAME

docs/pdds/pddxx_language_interop.pod - Inter-language calling

=head1 VERSION

$Revision: 28231 $

=head1 ABSTRACT

This PDD describes Parrot's conventions and support for communication between
high-level languages (HLLs).  It is focused mostly on what implementors should
do in order to provide this capability to their users.

=head1 DESCRIPTION

The ability to mix different high-level languages at runtime has always been
an important design goal of Parrot.  Another important goal, that of
supporting all dynamic languages, makes language interoperability especially
interesting -- where interesting means the same as it does in the Chinese
curse, May you live in interesting times.  It is expected that language
implementers, package authors, and package users will have to be aware of
language boundaries when writing their code.  It is hoped that this will not
become too burdensome.

None of what follows is binding on language implementors, who may do whatever
they please.  Nevertheless, we hope they will at least follow the spirit of
this document so that the code they produce can be used by the rest of the
Parrot community, and save the fancy footwork for intra-language calling.
However, this PDD Bis binding on Parrot implementors, who must provide a
stable platform for language interoperability to the language implementors.

=head2 Ground rules

In order to avoid N**2 complexity and the resulting coordination headaches,
each language compiler provides an interface as a target for other languages
that should be designed to require a minimum of translation.  In the general
case, some translation may be required by both the calling language and the
called language:

|
|
|Calling sub
| |
|   Language X|
| V
|Calling stub
+ |
  |
  plain Parrot  |
  |
+ |
| V
|   Called wrapper
| |
| |
|   Language YV
| Called sub
|

Where necessary, a language may need to provide a wrapper sub to interface
external calls to the language's internal calling and data representation
requirements.  Such wrappers are free to do whatever translation is required.

Similarly, the caller may need to emit a stub that converts an internal call
into something more generic.

{{ Of course, stub is really too close to sub, so we should find a better
word.  Doesn't the C community call these bounce routines?  Or something?
-- rgr, 31-Jul-08. }}

{{ I am discovering that there are five different viewpoints here,
corresponding to the five layers (including plain Parrot) of the diagram
above.  I need to make these viewpoints clearer, and describe the
responsibilities of each of these parties to each other.  -- rgr,
31-Jul-08. }}

Languages are free to implement the stub and wrapper layers (collectively
called glue) as they see fit.  In particular, they may be inlined in the
caller, or integral to the callee.

Ideally, of course, the plain Parrot layer will be close enough to the
semantics of both languages that glue code is unnecesary, and the call can be
made directly.  Language implementors are encouraged to dispense with glue
whenever possible, even if glue is sometimes required for the general case.

In summary:

=over 4

=item *

Each HLL gets its own namespace subtree, within which Cget_hll_global and
Cset_hll_global operate.  In order to make external calls, the HLL must
provide a means of identifying the language, the function, and enough
information about the arguments and return values for the calling language to
generate the call correctly.  This is necessarily language-dependent, and is
beyond the scope of this document.

=item *

When calling across languages, both the caller and the callee should try to
use plain Parrot semantics to the extent possible.  This is explained in
more detail below, but essentially means to use the simplest calling
conventions and PMC classes possible.  Ideally, if an API uses only PMCs that
are provided by a bare Parrot (i.e. one without any HLL runtime code), then
it should be possible to use this API from any other language.

=item *

It 

Re: [perl #57190] HLL Interoperation

[chromatic]

On Tuesday 22 July 2008 06:34:13 Moritz Lenz wrote:

 There are actually many problems. For example, if a TCL function returns
 an integer, what will Perl 6 see?

A PMC, which presumably performs the Integer role (in Parrot terms).  If Tcl 
on Parrot doesn't return this, Tcl on Parrot is wrong and bad.

This is why we have PMCs and PMC roles (okay, why we will have PMC roles) and 
especially why we have vtables.

If Tcl decides to reimplement its own data structures and not use vtables and 
PMC roles in the same way as other languages, it will interoperate about as 
well as the prototypical ugly American tourist complaining about Parisian 
traffic on Bastille Day.

I have trouble imagining that any other approach to interoperability works any 
better (and devolves into lesser utter chaos -- maybe I can get a hotel named 
after me for proving the existence of infinite forms of utter chaos).

The fact that tie() works in Perl 5 suggests that this approach works in 
practice.  Where's the problem?

-- c

Re: [perl #57190] HLL Interoperation

[Will Coleda]

On Wed, Jul 23, 2008 at 3:09 AM, chromatic [EMAIL PROTECTED] wrote:
 On Tuesday 22 July 2008 06:34:13 Moritz Lenz wrote:

 There are actually many problems. For example, if a TCL function returns
 an integer, what will Perl 6 see?

 A PMC, which presumably performs the Integer role (in Parrot terms).  If Tcl
 on Parrot doesn't return this, Tcl on Parrot is wrong and bad.

 This is why we have PMCs and PMC roles (okay, why we will have PMC roles) and
 especially why we have vtables.

There's more to PMCs than vtables. Is it reasonable for Tcl to always
expect that a string-like vtable will provide a get_list METHOD? If
not, how do we address that? What's the proper way to be able to add
methods to objects in a way that works in a cross-HLL platform? There
are vtables for some basic math, but how many HLLs are using the inc
vtable when someone calls $a+1?

Even simpler, the vtables obviously have different behaviors: Let's
say someone passes a perl string into a tcl function that calls [expr
$var++] and then proceeds to do some further usage of that value,
knowing that it's obviously safe because if it wasn't an integer, it
would have thrown an error on the ++. Except that because we're
operating on someone else's PMC there,  it -was- safe, so something
blows up unexpectedly later.

So, there's -some- language specific information in the PMCs. There's
some in the compiler/interpreter. Where is the proper boundary? -Is-
there a proper boundary?

Basically, we need to declare what folks expect is going to happen in
the cases where someone invokes things across HLL boundaries. If
parrot's policy is unless you do something stupid, it should work,
then we need to write down what those stupid things are so that those
of us implementing the HLLs can know to avoid them if we want to play
nice.

So, I'm curious: are METHODs in this category? I have functionality on
Tcl's PMCs in METHODS that I require for proper operation. I don't
check to see if the method exists before I invoke it. If it wasn't
there, I'm not sure what my fallback would be.

 If Tcl decides to reimplement its own data structures and not use vtables and
 PMC roles in the same way as other languages, it will interoperate about as
 well as the prototypical ugly American tourist complaining about Parisian
 traffic on Bastille Day.

Can you be more specific about how we could be using PMC roles to
improve our HLL interoperability? SFAIK these are defined only in
pmc2c POD with no actual code behind them.

 I have trouble imagining that any other approach to interoperability works any
 better (and devolves into lesser utter chaos -- maybe I can get a hotel named
 after me for proving the existence of infinite forms of utter chaos).

 The fact that tie() works in Perl 5 suggests that this approach works in
 practice.  Where's the problem?

 -- c

-- 
Will Coke Coleda

Re: [perl #57190] HLL Interoperation

[Bob Rogers]

   From: Geoffrey Broadwell [EMAIL PROTECTED]
   Date: Tue, 22 Jul 2008 22:00:42 -0700

   On Tue, 2008-07-22 at 22:58 -0400, Bob Rogers wrote:
   So I would argue that (1) what seem like differences in numbers in
the various languages are really differences in the way those languages
define their numeric operators, not in the numbers themselves;

   I disagree.  How do you represent Complex in a language that doesn't
   have a way to represent a number with more than one dimension?  This is
   a fundamentally different kind of thing than any simpler numeric type.

True.  But passing a Complex to any language that does not have a
concept of Complex is going to cause problems if the language tries to
treat it as anything but a black box.  And a black box doesn't require a
special representation.

   Granted, some conversion may be necessary when calling into a
language with a more limited complex type.  For example, Common Lisp
distinguishes (complex float), where Re and Im are both floating point,
from (complex rational), where they are both Integer, BigInt, or Ratio.
But most other languages don't.  In that case, mapping needs to happen,
and ought to be CL's responsibility when it compiles the call.

   But if you can't represent X in a language *at all*, what does it
mean to map such a thing?

 and (2)
standardizing on common numeric data type will avoid the impossible job
of making the Parrot built-in arithmetic be all things to all languages
(and all combinations thereof).

   That's certainly a possible choice, but it's still a mapping, and I
   argue not the only sane one.  More than one language has a Complex type,
   but not all of them do.  If we make Complex the base Parrot type that
   everything gets converted to, then some languages will be ... unhappy.
   If we standardize on some other numeric type, than we fail the round
   trip test spectacularly.

   I would further argue that *any* sort of type mapping is problematic
when calling across language boundaries.  If I pass an array of arrays
of ... of numbers from one language to another, then mapping would seem
to require a deep copy.  This changes the API:  The native call into
this routine can see side effects to the passed data structure, whereas
the foreign call would not.  (Or am I misunderstanding what you mean by
mapping here?)

   I have two answers to that:

 1. It may not be a deep copy; it may be an autobox of some sort.  But
that just begs the question of how to explain the source language's
semantics to all destination languages.  Something we need to
discuss, clearly.  :-)

Yes, indeed (to the question-begging part, especially.  ;-)

 2. We may simply decide that when you pass containers between HLLs,
you explicitely give up some or all of the source language's
guarantees, and the source language compiler is required to treat
the data structure after that point as if it had been exposed to
hard radiation.

   -'f

What guarantees?  When you pass a hash to function that expects that arg
to be an array, strange things are bound to happen.  And they are bound
to be equally strange regardless of whether the call to this function
was foreign or native.  It seems strange to expect inter-language
calling to be somehow safer than intra-language calling.

-- Bob

P.S.  I will be incommunicado for a few days, so please don't interpret
silence as lack of interest.

Re: [perl #57190] HLL Interoperation

[Will Coleda]

On Wed, Jul 23, 2008 at 10:04 AM, Will Coleda [EMAIL PROTECTED] wrote:
 On Wed, Jul 23, 2008 at 3:09 AM, chromatic [EMAIL PROTECTED] wrote:
 On Tuesday 22 July 2008 06:34:13 Moritz Lenz wrote:

 There are actually many problems. For example, if a TCL function returns
 an integer, what will Perl 6 see?

 A PMC, which presumably performs the Integer role (in Parrot terms).  If Tcl
 on Parrot doesn't return this, Tcl on Parrot is wrong and bad.

 This is why we have PMCs and PMC roles (okay, why we will have PMC roles) and
 especially why we have vtables.

 There's more to PMCs than vtables. Is it reasonable for Tcl to always
 expect that a string-like vtable will provide a get_list METHOD? If
 not, how do we address that? What's the proper way to be able to add
 methods to objects in a way that works in a cross-HLL platform? There
 are vtables for some basic math, but how many HLLs are using the inc
 vtable when someone calls $a+1?

 Even simpler, the vtables obviously have different behaviors: Let's
 say someone passes a perl string into a tcl function that calls [expr
 $var++] and then proceeds to do some further usage of that value,
 knowing that it's obviously safe because if it wasn't an integer, it
 would have thrown an error on the ++. Except that because we're
 operating on someone else's PMC there,  it -was- safe, so something
 blows up unexpectedly later.

One more slightly complicated example. Tcl has two different hash-like
values. One is manipulated using [dict] and friends, and can have
nested dictionaries, and is representable as a string. The other is
manipulated using [array] and friends, can only have a single top
level set of keys, and cannot be represented as a string. Both are
delcared as being Hash-y in their PMC declaration. But in this case, a
lot of the functionality for dealing with them is in the commands
themselves because of the language. In this case parrot's set of roles
is insufficient for me to make a determination as to which type I
might conceivable want to pass into another HLL (Takes a hash? ok,
have a TclArray!), and if I get something Hash-like and someone tries
to use 'array' on it... do we have any reasonable expectations of what
should happen? What if someone passes me a perl Hash to something that
tries to call Array? Right now I explicitly check the type of the PMC
to verify I can work on it. This isn't really a portable way to do
this, but what is?

Part of our problem is that HLL interaction doesn't really work for
anyone at the moment, so we can't write you a test case that blows up
and say here, fix this or tell us why.

 So, there's -some- language specific information in the PMCs. There's
 some in the compiler/interpreter. Where is the proper boundary? -Is-
 there a proper boundary?

 Basically, we need to declare what folks expect is going to happen in
 the cases where someone invokes things across HLL boundaries. If
 parrot's policy is unless you do something stupid, it should work,
 then we need to write down what those stupid things are so that those
 of us implementing the HLLs can know to avoid them if we want to play
 nice.

 So, I'm curious: are METHODs in this category? I have functionality on
 Tcl's PMCs in METHODS that I require for proper operation. I don't
 check to see if the method exists before I invoke it. If it wasn't
 there, I'm not sure what my fallback would be.

 If Tcl decides to reimplement its own data structures and not use vtables and
 PMC roles in the same way as other languages, it will interoperate about as
 well as the prototypical ugly American tourist complaining about Parisian
 traffic on Bastille Day.

 Can you be more specific about how we could be using PMC roles to
 improve our HLL interoperability? SFAIK these are defined only in
 pmc2c POD with no actual code behind them.

 I have trouble imagining that any other approach to interoperability works 
 any
 better (and devolves into lesser utter chaos -- maybe I can get a hotel named
 after me for proving the existence of infinite forms of utter chaos).

 The fact that tie() works in Perl 5 suggests that this approach works in
 practice.  Where's the problem?

 -- c

 --
 Will Coke Coleda




-- 
Will Coke Coleda

Re: [perl #57190] HLL Interoperation

[Geoffrey Broadwell]

On Wed, 2008-07-23 at 10:11 -0400, Bob Rogers wrote:
 True.  But passing a Complex to any language that does not have a
 concept of Complex is going to cause problems if the language tries to
 treat it as anything but a black box.  And a black box doesn't require a
 special representation.

But we *do* need to have a defined way to pass black boxes back and
forth, as when registering (and later calling) a cross-HLL callback with
associated data.

But if you can't represent X in a language *at all*, what does it
 mean to map such a thing?

That's a good question.  Does it throw an exception?  Does it
automatically become a black box?  Does it convert to an opaque object
with methods?  Does it become a frozen data structure that the
destination language just doesn't know how to thaw?

Defining behavior for the exceptional cases must be part of our type
mapping system.

  2. We may simply decide that when you pass containers between HLLs,
   you explicitely give up some or all of the source language's
   guarantees, and the source language compiler is required to treat
   the data structure after that point as if it had been exposed to
   hard radiation.
 What guarantees?  When you pass a hash to function that expects that arg
 to be an array, strange things are bound to happen.  And they are bound
 to be equally strange regardless of whether the call to this function
 was foreign or native.  It seems strange to expect inter-language
 calling to be somehow safer than intra-language calling.

An inter-language call could be less safe or more safe, depending on
implementation.  If Parrot guarantees that a deep copy will happen, or
that the structure passed is somehow guaranteed to be read-only, there
may be less danger than even a native call.  If Parrot *might* pass a
writeable reference to the original structure, then the source language
needs to treat an inter-HLL call as Tainting not just the structure
contents, but the structure itself.


-'f

Re: [perl #57190] HLL Interoperation

[Daniel Ruoso]

Qua, 2008-07-23 às 00:09 -0700, chromatic escreveu:
 On Tuesday 22 July 2008 06:34:13 Moritz Lenz wrote:
  There are actually many problems. For example, if a TCL function returns
  an integer, what will Perl 6 see?
 A PMC, which presumably performs the Integer role (in Parrot terms).  If Tcl 
 on Parrot doesn't return this, Tcl on Parrot is wrong and bad.
 This is why we have PMCs and PMC roles (okay, why we will have PMC roles) and 
 especially why we have vtables.

There are two important aspects on this discussion:

The first is that this collide with the representation polymorphism
feature of Perl 6. You can only assume to know the internal
representation (PMC with a Integer role) for native types. For a
non-native type, I might have an ad-hoc object that answers true
to .^does(Int), and the interpreter should behave accordingly.

The second is that even if you solve that for Integer, how do you solve
that for java.util.Vector, the Java List API (and all the
implementations) and the Collection API (and all the implementations)
(and all the other zillions of different APIs Java has)? Do you imply
that you'll have to re-implement the Java core API in terms of PMCs?

Even then, if some ad-hoc Java Class implements java.util.Collection,
how will you represent that in terms of a List PMC?...

daniel

[perl #57190] HLL Interoperation

[via RT]

# New Ticket Created by  Moritz Lenz 
# Please include the string:  [perl #57190]
# in the subject line of all future correspondence about this issue. 
# URL: http://rt.perl.org/rt3/Ticket/Display.html?id=57190 


Recently on IRC the matter of HLL interoperation came up frequently,
this ticket brings it to RT and the list.

Motivation:
It should be possible to use libraries written in a different HLL, for
example you might want to call a TCL library from Perl 6.

Problem:
There are actually many problems. For example, if a TCL function returns
an integer, what will Perl 6 see? A Perl 6-native Int? or a
Int:lang(TCL) (or whatever the syntax for a type name from a different
language is)? How will this conversion be made?
Can it be done without a separate bridge for each pair of HLLs?
And finally, it needs to be documented.

For a few notes, and pointers to the discussions on IRC please see
http://www.perlfoundation.org/parrot/index.cgi?intra_hll_mapping_notes


To not create another unclosable ticket, I consider this ticket closed when
1) there is a proof-of-concept implementation that allows you to call
functions/methods from a different HLL, and meaningfully work with its
results in one of the major languages (ie not a language built
specifically for that purpose)
2) There is documentation on how to make your language cross-HLL-ready,
either in PCT or core documentation.

Cheers,
Moritz

-- 
Moritz Lenz
http://moritz.faui2k3.org/ |  http://perl-6.de/

[perl #57190] HLL Interoperation

[Bob Rogers]

   From: Moritz Lenz (via RT) [EMAIL PROTECTED]
   Date: Tue, 22 Jul 2008 06:34:13 -0700

   Recently on IRC the matter of HLL interoperation came up frequently,
   this ticket brings it to RT and the list.

   Motivation:
   It should be possible to use libraries written in a different HLL, for
   example you might want to call a TCL library from Perl 6.

   Problem:
   There are actually many problems. For example, if a TCL function returns
   an integer, what will Perl 6 see? A Perl 6-native Int? or a
   Int:lang(TCL) (or whatever the syntax for a type name from a different
   language is)? 

Small plea:  For numbers at least, and preferably for anything else
which can be described in mathematical terms, Parrot ought to have as
small a set of representation types as possible, in order to minimize
the need for mapping (if not eliminate it entirely) when calling across
language boundaries.

   Start with something simple, like the meaning of the integer three
divided by the integer five, which differs significantly between
languages:

   In Tcl, this division of two integers produces the integer value 0.

   In Perl 5 and Lua, this division produces the floating-point value
0.6.

   In Common Lisp, this division produces 3/5, the mathematically-
exact ratio with numerator 3 and denominator 5.

   One way to implement these different meanings is to provide a common
division operator that dispatches on the types of its operands.  This
fails for operands of mixed type; dividing a Perl integer by a Lua
integer is straightforward, but what should we do for division of a Tcl
integer by a Lisp integer?  Or vice versa?

   An even bigger problem is that no Perl 5 code, when given two
integers to divide, will expect a Common Lisp ratio as a result.  Any
Perl 5 implementation that does this has a bug, even if both those
integers happen to come from Common Lisp.  Ditto for a floating-point
result from Common Lisp code that happens to get two integers from Perl
or Lua (or both!).

   So I would argue that (1) what seem like differences in numbers in
the various languages are really differences in the way those languages
define their numeric operators, not in the numbers themselves; and (2)
standardizing on common numeric data type will avoid the impossible job
of making the Parrot built-in arithmetic be all things to all languages
(and all combinations thereof).

   I would further argue that *any* sort of type mapping is problematic
when calling across language boundaries.  If I pass an array of arrays
of ... of numbers from one language to another, then mapping would seem
to require a deep copy.  This changes the API:  The native call into
this routine can see side effects to the passed data structure, whereas
the foreign call would not.  (Or am I misunderstanding what you mean by
mapping here?)

-- Bob Rogers
   http://rgrjr.dyndns.org/

Re: [perl #57190] HLL Interoperation

[Geoffrey Broadwell]

On Tue, 2008-07-22 at 22:58 -0400, Bob Rogers wrote:
So I would argue that (1) what seem like differences in numbers in
 the various languages are really differences in the way those languages
 define their numeric operators, not in the numbers themselves;

I disagree.  How do you represent Complex in a language that doesn't
have a way to represent a number with more than one dimension?  This is
a fundamentally different kind of thing than any simpler numeric type.

  and (2)
 standardizing on common numeric data type will avoid the impossible job
 of making the Parrot built-in arithmetic be all things to all languages
 (and all combinations thereof).

That's certainly a possible choice, but it's still a mapping, and I
argue not the only sane one.  More than one language has a Complex type,
but not all of them do.  If we make Complex the base Parrot type that
everything gets converted to, then some languages will be ... unhappy.
If we standardize on some other numeric type, than we fail the round
trip test spectacularly.

I would further argue that *any* sort of type mapping is problematic
 when calling across language boundaries.  If I pass an array of arrays
 of ... of numbers from one language to another, then mapping would seem
 to require a deep copy.  This changes the API:  The native call into
 this routine can see side effects to the passed data structure, whereas
 the foreign call would not.  (Or am I misunderstanding what you mean by
 mapping here?)

I have two answers to that:

  1. It may not be a deep copy; it may be an autobox of some sort.  But
 that just begs the question of how to explain the source language's
 semantics to all destination languages.  Something we need to
 discuss, clearly.  :-)

  2. We may simply decide that when you pass containers between HLLs,
 you explicitely give up some or all of the source language's
 guarantees, and the source language compiler is required to treat
 the data structure after that point as if it had been exposed to
 hard radiation.


-'f