Re: C# (.NET) has no interpreters

[John Tobey]

Simon Peyton-Jones <[EMAIL PROTECTED]> wrote:
> You can't really implement C-- on top of C efficiently, because of
> (a) tail calls and (b) the runtime interface for garbage collection,
> exception handling etc.

Agreed.  But any practical C-- implementation will start with a C/C++
compiler so that it can get structs and typedefs right, and preferably
also inline functions.  It would have to discriminate and reject
"ill-behaved" C constructs.

I was not thinking of implementing C-- "on top of" C; certainly not on
top of ANSI C.  But I could imagine a (Perl?) program that translates
C-- into asm-ified C without C function calls.  All you'd really need
are a few primitives for getting and setting the sp and ip, a list of
the registers, and some macros for inserting span metadata into a
different linker section.  All of this would be straightforwardly
implemented with GCC using a smallish machine-specific header.

I'll bet a non-optimizing version could be produced with no asm - just
use volatile for variables and goto for jumps.  Accessing the stack
pointer and inserting span data might require asm.

Best
-John

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

help writing RFC

[John Tobey]

Hi guys,

I'm in the middle of writing an RFC regarding Perl's use of the C
stack.  If any of you has the time, please take a look at what I've
written below and send me any comments or corrections.  I haven't got
to the implementation section yet, but it will involve smallish,
fixed-size stack frames for recursive runops(), a per-runlevel SV
arena, and judicious use of C.

Thanks
-John

=head1 TITLE

Binary stack layout to support accurate garbage collection and
continuations

=head1 VERSION

=head1 ABSTRACT

C is neither designed nor well suited for high-level language
interpreters or compiler output.  However, it can be bludgeoned into
allowing enough stack inspection to implement accurate mark-and-sweep
garbage collection and Lisp/Scheme-style continuations.  Perl might
gain performance from the former and a desirable feature in the
latter.

=head1 DESCRIPTION

=head2 Perl 5 Stack Management

Perl's virtual machine makes limited use of the machine stack as it
runs Perl code.  That is, it normally is not very deep in C function
calls.  This is a good thing, because C stack frames are a nuisance to
high-level languages.  This aspect of Perl 5 is somewhat formalized in
its C and C structures.

A C (``stack info'') structure is allocated whenever an op
must recursively call back into Perl.  Such calls occur in C,
object destructors, tied variable accessor methods, overloaded
operator invocations, signal handlers (including C<$SIG{__WARN__}> and
C<$SIG{__DIE__}>), and the Perl API function C.  There is
always a current, innermost C structure called
C, and the outer ones are attached to it in a linked
list.

A C (``jump environment'') structure is allocated when
I C code, such as an XSub or embedding program, wishes to
enter the Perl interpreter and guarantee that control returns to the
point of entry.  If no C is used, non-local jump operations
like C cause control to be transfered directly to an outer
C or equivalent frame, or they make the process terminate.

C structures are not especially useful outside of Perl, and
neither they nor Cs are part of the public API.  A C
is used by API functions such as C, C,
C, and C.  The C family of API
functions lets one request that a C be used by specifying
C in the flags.

=head2 Accurate garbage collection, continuations: impossible?

A tricky problem for garbage collection is to find live pointers in
the stack.  C makes this difficult by hiding the details of stack
layout behind its support for functions and local (``automatic'')
variables.  Typically, interpreters written in C maintain expensive,
global lists of objects to be released when a stack frame is unwound
(i.e., discarded through function return or an exception).  The Perl 5
API provides the macros C, C, C, and
C, which serve this purpose (among others).

If the unwinding code knew where in the machine stack to find the Perl
temporaries, the global lists could be jettisoned.  This might benefit
reference-counting garbage collection as well as the proposed new
mark-and-sweep systems.

What's more, being able to correlate machine stack addresses with Perl
values would facilitate the introduction of I to Perl.
[INCOMPLETE]

Re: RFC 35 / Re: perl6-internals-gc sublist

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> > ``Abundant macros, inline functions, and API functions will be
> > used wherever possible so as to allow fundamental changes to the
> > internal representation.''
> 
> That belongs in a different RFC, the one roughing out the embedding, 
> extension, and opcode APIs. (I've got those half-done too... :)

Just as long as it gets mentioned somewhere, I'll be happy.  :-)

> I'm starting to lean to a threaded/unthreaded pair of vtables for each 
> variable type, so you only call the locking code once a variable's been 
> explicitly shared (and therefore has a valid threading vtable as well as 
> presumably a valid sync structure)

That sounds like a good idea to me.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: RFC 35 / Re: perl6-internals-gc sublist

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 08:33 AM 8/6/00 -0400, John Tobey wrote:
> >Dan Sugalski <[EMAIL PROTECTED]> wrote:
> > > At 11:44 PM 8/5/00 -0400, Chaim Frenkel wrote:
> > > >Why waste space on flags? The vtbl should handle that.
> > >
> > > Because some flags are universal, and if they're not, this field
> > > gets tossed.
> >
> >Flags that vary only when the vptr varies could go in a field of the
> >vtable for speedy access.  I'd like to see us try to avoid a flags
> >element in the Perl 6 SV.
> 
> I'm not that worried about it at the moment, certainly not for space 
> reasons. Trying to avoid a flag field's OK, but I'm not sure how far its 
> worth going.

Would you consider adding something like this to the RFC?

``Abundant macros, inline functions, and API functions will be
used wherever possible so as to allow fundamental changes to the
internal representation.''

> >If the stuff lives in an arena, have one mutex-init lock per page of
> >the arena.
> 
> Gah! No! Bad! That can lead you to odd deadlock situations that are 
> damnably hard to debug. Locking an arena page means different threads 
> working on unrelated data can run into each other. It's sort of the moral 
> equivalent of unconditionally locking all the variables used in a sub every 
> time you enter that sub.

Sorry, I fail to see why.  To upgrade or destroy an object:

1. lock the object
2. lock the source arena
3. lock the destination arena
4. move bits
5. unlock the destination arena
6. unlock the source arena
7. unlock the object

SVs are never downgraded, so no one's source and destination are
another's respective destination and source.  Maybe the above sequence
isn't exactly right, but if we adhere to strict rules for lock
sequencing, there won't be a deadlock, right?

> Threads and locking is tricky enough that I'd like us to be really 
> conservative about it until we actually need to get more daring, so I'd 
> prefer to be as correct as possible.

Agreed.

> >This avoids requiring non-shared values to contain individual mutexen.
> 
> I expect anything could be shared at any point. Perl's flexible enough 
> about other things, it ought to be flexible about this, too.

Hmm, yes.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: RFC 35 / Re: perl6-internals-gc sublist

[John Tobey]

On Sun, Aug 06, 2000 at 08:33:52AM -0400, John Tobey wrote:
> If the stuff lives in an arena, have one mutex-init lock per page of
> the arena.  I imagine every arena page will be aligned on 4k or so and
> will have a structure at the beginning containing the interpreter
> pointer and stuff like this mutex.  This structure will be found by
> masking the final bits of the variable_data pointer.
> 
> struct arena_page_header {
> Interpreter* owner;
> Lock share_init_lock;
> /* ... ? */
> };
> 
> struct arena_page_header* sv_get_page_header(SV* sv) {
> return (struct arena_page_header*)sv->variable_data & ~0xfff;
> }
> 
> void sv_init_mutex(SV* sv) {
> Lock* init_lock = &sv_get_page_header(sv)->share_init_lock;
> lock(init_lock);
> sv_upgrade_to_shared(sv);
> /* sv_lock(sv); */
> unlock(init_lock);
> }

*sigh* I need more sleep.  sv_lock(sv) here is misplaced; sv_upgrade
needs to hold the lock that it creates.  This share_init_lock would be
better named simply C, since it's the same one that will be used
for moving data around, allocating and deallocating in the arena.

-John

> This avoids requiring non-shared values to contain individual mutexen.

Re: RFC 35 / Re: perl6-internals-gc sublist

[John Tobey]

On Sun, Aug 06, 2000 at 08:40:51AM -0400, John Tobey wrote:
> Chaim Frenkel <[EMAIL PROTECTED]> wrote:
> > >> If the sync data is in the SV, I believe there is a race
> > >> condition between the destruction and grabbing a lock.
> > 
> > DS> Nope. If the variable is shared, the lock will need to be taken
> > DS> to destroy it. If it's not shared it's not an issue.
> > 
> > How can you grab the lock if the data is destroyed? I think the
> > locking structure has to be outside the thing being locked.
> 
> Umm, you lock the data, then you destroy it.  Destruction is just like
> any other modification, except that there's nothing left to unlock
> when you're done.  Hmm, how do we politely tell the guy waiting for
> the lock we destroyed to have better luck next time...  Maybe don't
> free the memory until we lock the arena page.

Strike this last sentence, it's irrelevant.  (We do have to lock the
arena when freeing data, but not for this reason.)

Sorry.
-John

Re: RFC 35 / Re: perl6-internals-gc sublist

[John Tobey]

Chaim Frenkel <[EMAIL PROTECTED]> wrote:
> >> If the sync data is in the SV, I believe there is a race
> >> condition between the destruction and grabbing a lock.
> 
> DS> Nope. If the variable is shared, the lock will need to be taken
> DS> to destroy it. If it's not shared it's not an issue.
> 
> How can you grab the lock if the data is destroyed? I think the
> locking structure has to be outside the thing being locked.

Umm, you lock the data, then you destroy it.  Destruction is just like
any other modification, except that there's nothing left to unlock
when you're done.  Hmm, how do we politely tell the guy waiting for
the lock we destroyed to have better luck next time...  Maybe don't
free the memory until we lock the arena page.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: RFC 35 / Re: perl6-internals-gc sublist

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 11:44 PM 8/5/00 -0400, Chaim Frenkel wrote:
> >Why waste space on flags? The vtbl should handle that.
> 
> Because some flags are universal, and if they're not, this field
> gets tossed.

Flags that vary only when the vptr varies could go in a field of the
vtable for speedy access.  I'd like to see us try to avoid a flags
element in the Perl 6 SV.

> >Put the sync data into a seperate area and then access the external data.
> 
> There lies race conditions. (cf 5.6.0's lock issues) For synchronization 
> data to be valid it must be there at creation time. If it needs to be 
> tacked on later you run into race issues with multiple threads trying to 
> init the sync data.

If the stuff lives in an arena, have one mutex-init lock per page of
the arena.  I imagine every arena page will be aligned on 4k or so and
will have a structure at the beginning containing the interpreter
pointer and stuff like this mutex.  This structure will be found by
masking the final bits of the variable_data pointer.

struct arena_page_header {
Interpreter* owner;
Lock share_init_lock;
/* ... ? */
};

struct arena_page_header* sv_get_page_header(SV* sv) {
return (struct arena_page_header*)sv->variable_data & ~0xfff;
}

void sv_init_mutex(SV* sv) {
Lock* init_lock = &sv_get_page_header(sv)->share_init_lock;
lock(init_lock);
sv_upgrade_to_shared(sv);
/* sv_lock(sv); */
unlock(init_lock);
}

This avoids requiring non-shared values to contain individual mutexen.

> >And why carry around IV/NV and ptr? Make it into a union, to allow
> >room.
> 
> Speed? Valid tradeoff, though. We can do it both ways to see what's
> better.
> 
> >The string/number duality should be handled by the vtbl. So a
> >string that is never accessed as a number, doesn't waste the
> >space. And numbers that are rarely accessed as a string save some
> >room. And as needed the vtbl can be promoted to a duality version
> >that maintains both.
> 
> I agree with this, though I'm pretty sure most scalars end up with
> multiple types. I'm up for either way, though--

Really?  I find Perl 5's IV/NV/PV/PVIV/PVNV contraption with the fake
structure offsets kind of charming.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: RFC: Foreign objects in perl

[John Tobey]

On Sun, Aug 06, 2000 at 01:24:10AM -0400, Dan Sugalski wrote:
> John Tobey wrote:
>  > Picture this.  A Lisp (or Java, ...) structure holds a Perl
>  > interpreter.  A Perl variable holds a reference to the Lisp structure.
>  > Structure and interpreter become inaccessible to all threads.  Perl
>  > will never know it's done with the Lisp structure, neither Perl nor
>  > the structure will ever be collected, and we will have defeated
>  > mark-and-sweep.
>
> FWIW, this isn't a case I'm worried about. If an embedding program doesn't
> explicitly destroy an interpreter, then its reasonable to assume that the
> interpreter's not dead and shouldn't be cleaned up after--if someone's
> leaking interpreters then, well, they've got bigger problems than the odd
> foreign object. :)

I'm not so sure I agree with this.  Other-language modules in a large
application might have good reason to use a private Perl interpreter
within an encapsulated interface.

Anyway, my point is valid for data structures other than
interpreters.  Lisp variable `perl-int' holds an interpreter, which
contains a by-Perl-unreferenced anonymous scalar, which contains a
Lisp structure, which contains a reference to the scalar.


   -
  |  | |   |
  |  Perl >| SCALAR(0x8335bfc) |
  |  | |   |   |
   |
   |   ^
   |   |
   V   |
  |-+--|--|
  | |  |  |
  | car | cdr |
  | | |
  |-+-|

If Perl uses reference-counting as its only garbage collection, this
loop will have to be broken explicitly.  But if we anticipate possibly
implementing mark-and-sweep, our mark routine should call Lisp's mark
routine on those Lisp objects we still reference, which would allow
those we don't to be collected.

-John

Re: RFC 46 (v1) Use features of portable, free compilers

[John Tobey]

Your have a powerful argument for supporting argument standards and
avoiding use of features that are ill-conceived or differently
specified by a standard.  I will add these points to the document.

However, I believe I still have a strong case for encouraging adoption
of the good features of GNU CC and Libc as new standards and getting
vendors to contribute their optimizations.  No one I know really wants
to maintain portability code such as Configure.  Our time could be
much better spent.  At this point we would be gravely mistaken to
ignore other platforms, but eventually we must move on, as we did when
we abandoned non-ANSI compilers.

Regards.
-John

Re: RFC 46 (v1) Use features of portable, free compilers

[John Tobey]

On Sat, Aug 05, 2000 at 10:02:23PM -0500, Jarkko Hietaniemi wrote:
> On Sat, Aug 05, 2000 at 07:25:41PM -0700, Russ Allbery wrote:
> > libraries and tools are better than the vendor counterparts; Sun's
> > compilers and linkers are considerably better than GNU's for SPARC C code,
> > for example.
> 
> Likewise for Digital/Compaq.  gcc's Alpha code generation and optimization
> is really inferior.

I don't doubt this in the least, but I still use GCC on SPARC systems
and would be reluctant to buy a system with a poor GCC port.  If you
want my business, work on the GCC port for your system, not some lousy
proprietary compiler.  It's not even about politics.  I'd take a good,
portable, proprietary compiler if it gave me all the benefits of GCC
and more.  Similary for linkers and libraries.

I am furthermore *not* suggesting dropping support for non-Unix or
non-GCC platforms.  I am merely saying that that is not where our
focus should be.

Cheers
-John

RFC - Use features of portable, free compilers and libraries

[John Tobey]

I'm sending this to perl6-internals and BCC'ing it to perl6-rfc.
Maybe that is a good compromise.

-John

=head1 TITLE

Use features of portable, free compilers and libraries

=head1 VERSION

  Maintainer: John Tobey <[EMAIL PROTECTED]>
  Date: 5 Aug 2000
  Version: 1
  Mailing List: [EMAIL PROTECTED]
  Number: 

=head1 ABSTRACT

There is no sane reason why *nix vendors continue to push proprietary
compilers and system libraries on their customers when better, free
replacements could be had for little effort.  Eventually, they will
realize this and start porting GNU Libc and Binutils, contributing
whatever unique features their current tools have to the GNU versions,
and shipping these packages with their systems.  Perl should take
aggressive advantage of these programs' features in anticipation of
eventually not having to support all the other cruft that's out there.

=head1 DESCRIPTION

Sun beats AIX and loses to Linux for (practically) no other reason
than the intermediate quality of its GCC and GNU Binutils ports.  Bla,
bla, Glibc is portable, so port it.  Get with it, Sun!  Get with it,
HP!  Wake up, IBM!  Come on, SGI!  Hello, MSFT!  The GNU stuff is
nice.  It is free.  If it comes with the system, there is no licensing
war.

Think about how much it would cost the vendors to merge their features
into the GNU stuff.  Then think about how much developer effort would
be saved worldwide if all major systems shared a C library, compiler,
preprocessor, and linker.  Configure would be 5% its current size, if
it would exist at all.

We now have the chance to B by giving these clowns a
serious incentive to do the B.

=head1 IMPLEMENTATION

Perl 5 already takes advantage of some GNU CC features.  It should
henceforth consider anything below GCC 2 as C.  Perl
works around limitations of Standard C library functions that GNU Libc
has already overcome.  C comes to mind.  C,
I somebody tell me I can use C!  We should stop
duplicating effort and take advantage of that code.  GNU CPP supports
varargs macros, so why not ship GNU CPP with perl?  There is no GPL
issue if it is only used in the build.

=head1 ISSUES

I have not looked at the BSD C library, so I don't know what to say
about it.  Is it portable?  Is it as featureful as GNU Libc?  Does it
perform as well or better?

=head1 REFERENCES

The GNU C Library Reference Manual, Using and Porting GNU CC, GNU
Binutils manual.  Get 'em through www.gnu.org.

Re: RFC 35 v1, internal base format for perl variables

[John Tobey]

Hmm, somehow this slipped through the cracks as I was changing list
subscription addresses.  Oh, well.

Below are my quickie, off-the-cuff reactions to Dan's RFC #35v1.
Recipe cum grano salis.

>The base variable structure looks like:
>
>struct {
>  IV GC_data;
>  void *variable_data;
>  IV flags;
>  void *vtable;
>  void *sync_data;
>}

Cool.

>The fields, in order, are:
>
>=over 4
>
>=item GC_data
>
>Random data for the garbage collector, whichever one is used. This
>could be a marker for M&S GC, or a refcount for refcount GC, or just
>nothing at all if we get a really clever GC.

Really clever, like M&S and using the bottom pointer bit as the
marker.  There is no need for an IV as marker.  Not to mention, that
if these structs are passed around by value instead of required to
live in some arena, they won't need *ANY* GC data.  Clever!

>=item variable_data
>
>Equivalent to perl5's sv_any pointer, this is a pointer to the actual
>data structure for the variable. It may, in certain cases, be coopted
>to hold the actual value.

Go Dan!

> (This is likely the case for a scalar that
>holds just an integer, where the native int size is equal to or
>smaller than the native pointer size)
>
>The actual structure that hangs off will depend both on the class of
>variable (scalar, hash, array) and the type of that class (integer
>array, integer scalar, filehandle, reference) and isn't specified here
>
>=item flags
>
>This field holds various flags that hold the status of the
>variable. (Flags to be RFC'd later)

Please RFC the flags pronto.  Maybe we don't need any.

>=item vtable
>
>The vtable field holds a pointer to the vtable for a variable. Each
>variable type has its own vtable, holding pointers to functions for
>the variable. Vtables are shared between variables of the same
>type. (All integer arrays have the same vtable, as do all string
>scalars and so on)
>
>vtable contents will be RFCd separately. All variables will share a
>common set of functions, though scalars, arrays, and hashes will have
>their own set of extensions on top of that.

Okay, vtables are yummy.  I'll take a vptr along with my
variable_data, please.  With a nice, big, fat vtable on the other end
and lots of sauce.

>=item sync_data
>
>A pointer to a structure used for synchronization between async parts
>of perl. This will probably be used mainly for threading, but may be
>used to indicate interpreter ownership for variables shared between
>interpreters, or other such things. If 0, the scalar is assumed to be
>unshared and no synchronization is done.

Interpreter ownership: Stuff that exists "by value" does not need an
owner.  Stuff that lives in an arena should be able to find its
interpreter with something like *(Interpreter**)(ptr & ~0x3ff).

Thread mutexing: only special fat shared_things can be mutexed.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: perl6-internals-gc sublist

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> That's going to complicate a lot of things, and it will mean perl will end 
> up doing all sorts of temp SV creation when these 'mini SVs' get passed to 
> subs. Plus it complicates life a lot for people writing the guts. (Which 
> would be us. :)

The proof is in the pudding, so I won't add more to this discussion
until I have some code to back it up.

> I think this sort of optimization is very premature,

We're not talking about optimizing (or even implementing) this today.
We're talking about leaving the door open.

> and it will require 
> jumping through a lot of hoops because of perl's profligate use of dynamic 
> polymorphing. If it's needed, well, we'll do it, but I'd rather put the 
> effort elsewhere to start.

Not to be rude, but the effort is not yours to put.  Except for your
own effort, that is.  Although I'll read your every suggestion with
due consideration, I (and, I imagine, most of us here) will code
exactly that which interests me.

Best
-John

Re: perl6-internals-gc sublist

[John Tobey]

Nick Ing-Simmons <[EMAIL PROTECTED]> wrote:
> John Tobey <[EMAIL PROTECTED]> writes:
> >Dan Sugalski <[EMAIL PROTECTED]> wrote:
> >> Yup, and I realized one of my big problems to GCs that move memory 
> >> (references that are pointers and such) really isn't, if we keep the 
> >> two-level variable structure that we have now. The 'main' SV structure 
> >> won't move, while the guts that the equivalent of sv_any points to can 
> >> without a problem.
> >
> >I certainly hope this data layout factoid is still subject to change.
> 
> Having an SV have a fixed address is handy for C extensions.
> The 'entity' has got to have some 'handle' to defines its existance.
> If not the SV* data structure then what is it that defines the thing?

Just like in Perl, if you want a reference to it, put it somewhere and
use a pointer.  Otherwise, use it by value.  (three words, or two if
flags are dropped, or four if vptr is added, or one if everything is
crammed into a pointer with low bits commandeered).

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: C--

[John Tobey]

John Tobey <[EMAIL PROTECTED]> wrote:
> Joshua N Pritikin <[EMAIL PROTECTED]> wrote:
> > A few more clicks and I found:
> > 
> >   http://www.cminusminus.org/
> 
> Thanks, Joshua.  Quickie summary.  Implementations: one[1] semi-free
> (non-DFSG-compliant) complete.  Others in progress.

My quickie summary seems to be inaccurate.  The lcc thing is not an
implementation, it's a C-- code generator (why??).

There is a complete implementation based on MLRISC, but it appears to
lack a license.  I guess that means it's free, but one should ask the
author for clarification.  MLRISC's own document claims that "MLRISC
is I software, and is released under the SML/NJ
license." (emphasis in original)  But the URL given for the license is
stale.

Not a Microsoft conspiracy?  I'll reserve judgment.  (<-- this is a
joke)

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: C--

[John Tobey]

Nick Ing-Simmons <[EMAIL PROTECTED]> wrote:
> John Tobey <[EMAIL PROTECTED]> writes:
> >Joshua N Pritikin <[EMAIL PROTECTED]> wrote:
> >> A few more clicks and I found:
> >> 
> >>   http://www.cminusminus.org/
> >
> >Thanks, Joshua.  Quickie summary.  Implementations: one[1] semi-free
> >(non-DFSG-compliant) complete.  Others in progress.
> >
> >Why not specify as a C extension: I'm still looking for that.
> 
> Could one do a GCC front end for C-- ?

I'm one step ahead of you.  Last night I started reading gcc/cp/lex.c
from the GCC source.  :-)  My idea is to introduce a

__cminusminus__ {
// ...
}

construct in the C and C++ syntaxes.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

relocatable perl, again (was Re: kpathsea)

[John Tobey]

[CC'd to perl6-build]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 03:25 PM 8/4/00 -0700, Russ Allbery wrote:
> >Simon Cozens <[EMAIL PROTECTED]> writes:
> >
> > > Sounds good. Here's a slight modification: perllib.db is a cache;
> > > lookups take place as normal, but then any new information is written
> > > into the cache.  The cache is invalided every $configurable period.
> >
> >Putting on my sysadmin hat, anything that a program wants to modify during
> >its normal course of operation and isn't a dotfile in the user's home
> >directory is inherently Evil and not to be tolerated if at all possible.
> 
> I fully agree. Not that I dislike the idea of figuring out file locations 
> at module install time instead of runtime, mind--I rather like that. (Do 
> the work once, rather than every darned time) make install, however, should 
> be the only thing that messes with these files.

How tough would this be?

# perl -MPerl::Mover -e 'copyperl "/home/me/local"'

Perl/Mover.pm does whatever's needed regarding binedits, relinking,
and whatnot.  Would this allow us to remove the discussion from
internals, at least?

-John

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: perl6-internals-gc sublist

[John Tobey]

OK.  Ask, cancel that request.  Sorry.
-John

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 02:45 PM 8/4/00 -0400, John Tobey wrote:
> >I guess, more than establishing a working group, I'm hoping to siphon
> >GC debates out of the more general internals list, since a lot of
> >people love discussing GC at great length, and I don't expect that
> >debate to have much relevance to the rest of the internals.
> 
> I'd rather not do that, though. The perl lists are supposed to be rather 
> more targeted than that. (Not that I mind a good GC discussion, but the 
> point is to get an RFC for a workable scheme)
> 
>   Dan
> 
> --"it's like this"---
> Dan Sugalski  even samurai
> [EMAIL PROTECTED] have teddy bears and even
>   teddy bears get drunk
> 
> 

Re: perl6-internals-gc sublist

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> Yup, and I realized one of my big problems to GCs that move memory 
> (references that are pointers and such) really isn't, if we keep the 
> two-level variable structure that we have now. The 'main' SV structure 
> won't move, while the guts that the equivalent of sv_any points to can 
> without a problem.

I certainly hope this data layout factoid is still subject to change.

-John

Re: perl6-internals-gc sublist

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> This seems a bit premature, given that we haven't actually come up with 
> even a framework for an API, or hashed out much on the format of variables 
> or the interpreter structure. (Nor the threading or event stuff...)
> 
> Are you comfortable with a shorter term list (Say,two or four weeks) that 
> will work up several different GC RFCs, based on various constraints? If 
> so, I'm fine with it.

Sure, if that's your preference.

I guess, more than establishing a working group, I'm hoping to siphon
GC debates out of the more general internals list, since a lot of
people love discussing GC at great length, and I don't expect that
debate to have much relevance to the rest of the internals.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: perl6-internals-gc sublist

[John Tobey]

Oops, I guess this should be contingent on Dan Sugalski's approval.
-John

John Tobey <[EMAIL PROTECTED]> wrote:
> Ask, can we please have the following list:
> 
> Name:   perl6-internals-gc
> Chairs: John Tobey <[EMAIL PROTECTED]>
> Ken Fox <[EMAIL PROTECTED]>
> Deadline:   Conterminous with perl6-internals (currently 13
>   October 2000)
> Mission:Determine which garbage collection models will be
>   supported.  Identify requirements and implications of
>   such support.
> Description:This list will discuss possible memory management
>   frameworks and requirements they would place on API,
>   language, and internals.
> 
> Thanks in advance,
> -John
> 
> 

perl6-internals-gc sublist

[John Tobey]

Ask, can we please have the following list:

Name:   perl6-internals-gc
Chairs: John Tobey <[EMAIL PROTECTED]>
Ken Fox <[EMAIL PROTECTED]>
Deadline:   Conterminous with perl6-internals (currently 13
October 2000)
Mission:Determine which garbage collection models will be
supported.  Identify requirements and implications of
such support.
Description:This list will discuss possible memory management
frameworks and requirements they would place on API,
language, and internals.

Thanks in advance,
-John

Re: RFC: Foreign objects in perl

[John Tobey]

Ken Fox <[EMAIL PROTECTED]> wrote:
> John Tobey wrote:
> > I think we are trying to accommodate any of several GC systems to be
> > selected amongst in future.
> 
> Then the Perl API needs to allow for the GC to move objects. If that
> can't happen, the majority of interesting collectors can't be used.

Yes, the Perl API needs to be extended for any of this to work.  At a
minimum, we'll need a Perl_mark(pTHX) or similar.

> > Ken Fox <[EMAIL PROTECTED]> wrote:
> > > Did you ever have this problem [cycles between systems] or is it
> > > contrived?
> > 
> > I personally guarantee the issue will arise.  :-)
> 
> Your finalizer doesn't catch cycles. Did you ever have problems with
> it not reclaiming garbage? A collector can still be useful even if it
> can't find all garbage. (Conservative collectors can't for example.)

Right, it won't catch cycles containing Perl structures, the same as
in Perl.

I suspect separating GC discussion out from this list will do more
good than harm, but in a 5-minute search I haven't found the formally
correct way (if there is one) to suggest a perl6-internals-gc list.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: RFC: Foreign objects in perl

[John Tobey]

I think we are trying to accommodate any of several GC systems to be
selected amongst in future.

Ken Fox <[EMAIL PROTECTED]> wrote:
> John Tobey wrote:
> > Picture this.  A Lisp (or Java, ...) structure holds a Perl
> > interpreter.  A Perl variable holds a reference to the Lisp structure.
> > Structure and interpreter become inaccessible to all threads.  Perl
> > will never know it's done with the Lisp structure, neither Perl nor
> > the structure will ever be collected, and we will have defeated
> > mark-and-sweep.
> 
> In the normal case, lisp runs the finalizer when the perl structure
> becomes garbage. In the finalizer, perl can erase the external
> reference and then next time perl collects, the structure may be
> marked as garbage. If you're talking about the situation where
> an object is only reachable through a pointer held by an external
> system, then lisp needs to use a two-phase finalization. First it
> tells the foreign system (perl) that the object is considered
> garbage except for the foreign reference. That allows perl to release
> the reference and then the object really does become garbage. If perl
> decides not to release, then the object remains a weak root. This is
> a really weird situation though and I wouldn't be surprised if most
> systems just ignore the possibility. Did you ever have this problem
> or is it contrived?

I personally guarantee the issue will arise.  :-)

> Foreign references are not swept and can't be swept unless you
> are building a gc system for non-cooperative environments (Boehm's
> collector for example) and they have a *lot* of constraints that
> most lisp systems I know of violate. The theoretical cycle won't
> be found.

Alas, I don't have a CS background.  But I have implemented garbage
collection glue between Perl and Lisp.  Here's my finalizer.  As far
as I know, it is safe and complete, but the code has never been
heavily tested, to my knowledge.  This implementation does not have
the concept of weak roots.  Every blessed Emacs::Lisp::Object holds a
mark slot within the perl-interpreter object, and perl-interpreter's
mark method marks them all.

/* Called from Fgarbage_collect.  */
static void
lisp_sv_destroy (object)
 Lisp_Object object;
{
  struct emacs_perl_interpreter *perl;
  SV *sv;

  /* Check the interpreter's status to avoid a crash after `perl-destruct'.
 During Emacs destruction, even this is dangerous.
 (The solution is to call `perl-destruct' *before* Emacs starts to
 destruct.)  */
  if (EQ (emacs_phase, Qdestructing))
return;

  perl = LISP_SV_INTERP (object);
  if (PERL_DEFUNCT_P (perl)
  || EQ (perl->phase, Qdestructing))
return;

  sv = (SV *) XFOREIGN_OBJECT (object)->data;

  if (SvREFCNT (sv) > 1 || ! perl->frame_p || ! SvROK (sv))
{
  SvREFCNT_dec (sv);
  return;
}
  call_perl (SV_FREE, sv);
}

-John

Re: ffcall GPL -> LGPL

[John Tobey]

Tom Christiansen <[EMAIL PROTECTED]> wrote:
> Does that mean that we're tossing out all other less onerous licensing
> schemata, like Artistic or BSD, and that consequently Perl is now
> guaranteed to be infested by the infinitely divisive political
> problems and pernicious peculations of rms and his hordes?

No, it's a CPAN module.

> I surely hope not.

Peace.

> --tom

Re: RFC: Foreign objects in perl

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 06:11 PM 8/2/00 -0400, John Tobey wrote:
> >Dan Sugalski <[EMAIL PROTECTED]> wrote:
> > > At 05:39 PM 8/2/00 -0400, John Tobey wrote:
> > > > > A scalar is made an object via a call into the perl
> > > > > library. The scalar is marked as an object and stuck into a
> > > > > package. Attached to the scalar is a pointer to the native
> > > > > object, a pointer to a generic dispatch routine, and a
> > > > > pointer to the native destruction routine for the object.
> > > >
> > > >Perhaps add a native mark routine, if we are really
> > > >GC-agnostic.
> > >
> > > When the object finally goes out of scope then we call the
> > > destroy method and assume that everything hanging off the object
> > > pointer is gone. That only leaves the perl bits to clean up, and
> > > we know about those.
> >
> >Don't assume that the Perl interpreter is the sole root of
> >accessibility.  Maybe some other language has 'us' (i.e., the
> >interpreter) as an object and needs to mark all its own objects
> >that we refer to for the benefit of its own GC.
> 
> Right, and I'm not. The destroy method is really the perl "I'm done
> with you" method. Perl calls it when the object falls out of scope
> and perl would call the cleanup code for it. The code that owns the
> object can then do whatever it needs to. The function should be
> renamed, I agree.

Picture this.  A Lisp (or Java, ...) structure holds a Perl
interpreter.  A Perl variable holds a reference to the Lisp structure.
Structure and interpreter become inaccessible to all threads.  Perl
will never know it's done with the Lisp structure, neither Perl nor
the structure will ever be collected, and we will have defeated
mark-and-sweep.

I still think foreign objects need a MARK method in addition to
DESTROY (or RELEASE, whatever).

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: RFC: Foreign objects in perl

[John Tobey]

Simon Cozens <[EMAIL PROTECTED]> wrote:
> On Thu, Aug 03, 2000 at 01:15:54PM -0400, Dan Sugalski wrote:
> > >One thing: remember, there is a lot of talk about having
> > >perl6 use Unicode internally, which means that things like
> > >method names should be wchar_t * (or whatever).
> > 
> > Good point. I shall have to think Unicode more. (UTF-32, anyone? :) Though 
> 
> Not necessary, but we should not rule it out. I think UTF16 internally is
> the way to go.
> 
> > it does present a problem for those languages that don't do Unicode and do 
> > char* instead.
>  
> I would guess we'd have to convert everything into Unicode.

What would be the trouble with having all relevant code (ops, SV
operations, what-have-you) compiled once for 8-bit chars, once for 16,
and once for 32?  The Unicode routines could be demand-loaded so as
not to affect programs that don't use them.

This is something like how the Windows 32 API works, I believe, where
FooBarBlahBlahBlah is a macro defined as either FooBarBlahBlahBlahA
(for "ASCII") or FooBarBlahBlahBlahW ("wide").

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: ffcall GPL -> LGPL

[John Tobey]

Garrett Goebel <[EMAIL PROTECTED]> wrote:
> With Bruno Haible's permission, I'm reposting a portion of our
> correspondence. The gist of which is that he'd be willing to make ffcall
> available for use with Perl under the LGPL.

Great.  I suggest that this topic be moved to the library list (or is
it "modules"?), as it does not affect the core, as far as I can see.

-John

Re: RFC: Foreign objects in perl

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> What, doing a
> 
>name = (PMC->vtable[NAME])(PMC);
> 
> is inflexible? I'm not talking about embedding data into the vtable, rather 
> make one of the table entries a pointer to a function that returns the type 
> name.

Oh, cool.  I misunderstood you.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Microsoft

[John Tobey]

Joshua N Pritikin <[EMAIL PROTECTED]> wrote:
> On Thu, Aug 03, 2000 at 11:30:40AM -0400, [EMAIL PROTECTED] wrote:
> > I'm not sure about this one.  My odds-on favorite answer: Picture some
> > M$ hackers telling their supervisor they are working on some GCC
> > enhancements.
> 
> But how do you explain the fact that S. P. Jones uses Latex and cygwin?

Easily.  He is the hacker, not the supervisor.

It's pretty evident to me that Microsoft and the free software coders
of the world have more in common than either has with, say, IBM.  The
licensing enforcement departments understand each other perhaps even
better than did Hitler and Stalin.  Some of the world's best coders
are in there, and the rest of us are out here.  :-)

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: RFC: Foreign objects in perl

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> > > Feel free to define some recommended functions. Give 'em all upper-case
> > > names so they don't get confused with real methods. (I hope) A PERL_ 
> > prefix
> > > might not be a bad thing either.
> >
> >The only one I feel strongly about here is TYPENAME, which should,
> >perhaps, be called by Perl's ref() builtin.  Hey-- then as a first
> >try, method calls could do a Perl fetchmeth()!  Wouldn't that spiff!
> 
> Yep. :) Make it one of the vtable entries--that way you can get a generic 
> perl magic cookie pointer and find out what the heck the thing really is.

Making it a data member of the vtable sacrifices extensibility.  I
started doing Perlmacs that way until I realized this.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: C--

[John Tobey]

Kevin Scott <[EMAIL PROTECTED]> wrote:
> John Tobey wrote:
> > 
> > Thanks, Joshua.  Quickie summary.  Implementations: one[1] semi-free
> > (non-DFSG-compliant) complete.  Others in progress.
> > 
> > Why not specify as a C extension: I'm still looking for that.
> > 
> > -John
> 
> Technical answer: C-- has lots of features that would be difficult to 
> hack into a C compiler.  Tail calls, continuations, good GC support,
> data layout control, and the list continues.  C also has lots of cruft
> that C-- doesn't need.

I disbelieve.

> Real answer: C-- is being designed and implemented by functional 
> programming guys who probably don't want to hack around in C compilers
> written in C.  Notice that the two prominent C-- implementations, 
> Quick C-- and Fermin Reig's C-- compiler are both written in functional
> programming languages: Quick C-- in Objective CAML, and Fermin's 
> compiler in Standard ML.

I'm not sure about this one.  My odds-on favorite answer: Picture some
M$ hackers telling their supervisor they are working on some GCC
enhancements.

-John

> -Kevin

Re: C--

[John Tobey]

Joshua N Pritikin <[EMAIL PROTECTED]> wrote:
> A few more clicks and I found:
> 
>   http://www.cminusminus.org/

Thanks, Joshua.  Quickie summary.  Implementations: one[1] semi-free
(non-DFSG-compliant) complete.  Others in progress.

Why not specify as a C extension: I'm still looking for that.

-John

[1] http://www.cs.princeton.edu/software/lcc/

Re: C# (.NET) has no interpreters

[John Tobey]

Joshua N Pritikin <[EMAIL PROTECTED]> wrote:
> On Wed, Aug 02, 2000 at 07:30:23PM -0400, [EMAIL PROTECTED] wrote:
> > Joshua N Pritikin wrote:
> > > perl5 is interpreter-centric with native code generation
> > > bolted on well into the development lifecycle.
> > 
> > I'd prefer us to tackle native code generation using C as the
> > intermediate language instead of a JIT.
> 
> Oh, yah.  C is the obvious choice, but it doesn't have to be the only
> backend.  In theory we could also generate C#'s IL.  Or C--.

Help.  I'm only halfway through the C-- paper, and I'm wondering:
What is the status of implmentations?  Why not implement it as
extensions to existing C compilers?

Thanks

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: C# (.NET) has no interpreters

[John Tobey]

> > While C might be fine and dandy for getting o.k. native code w/o too
> > much implementation effort, I think that it might be worth the effort
> > to implement a JIT compiler for the perl interpreter's intermediate
> > language.

Speaking of intermediate languages, is there any more concrete info on
the C# IL than what's in the article?  I'd like to know what would be
involved in supporting it.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: RFC: Foreign objects in perl

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 05:39 PM 8/2/00 -0400, John Tobey wrote:
> > > A scalar is made an object via a call into the perl library. The
> > > scalar is marked as an object and stuck into a package. Attached to
> > > the scalar is a pointer to the native object, a pointer to a generic
> > > dispatch routine, and a pointer to the native destruction routine for
> > > the object.
> >
> >Perhaps add a native mark routine, if we are really GC-agnostic.
> 
> When the object finally goes out of scope then we call the destroy method 
> and assume that everything hanging off the object pointer is gone. That 
> only leaves the perl bits to clean up, and we know about those.

Don't assume that the Perl interpreter is the sole root of
accessibility.  Maybe some other language has 'us' (i.e., the
interpreter) as an object and needs to mark all its own objects that
we refer to for the benefit of its own GC.

> >Native serializer?  Equality tester?  Type id accessor?  What prompts
> >me to propose these is the struct that evolved in Perlmacs for mixing
> >Perl and ELisp objects:
> 
> Feel free to define some recommended functions. Give 'em all upper-case 
> names so they don't get confused with real methods. (I hope) A PERL_ prefix 
> might not be a bad thing either.

The only one I feel strongly about here is TYPENAME, which should,
perhaps, be called by Perl's ref() builtin.  Hey-- then as a first
try, method calls could do a Perl fetchmeth()!  Wouldn't that spiff!

-John

> On the other hand, we may have a serialize (or equality) function as part 
> of the sv vtable, in which case the point's moot, sort of.
> 
>   Dan
> 
> --"it's like this"---
> Dan Sugalski  even samurai
> [EMAIL PROTECTED] have teddy bears and even
>   teddy bears get drunk
> 
> 

Re: RFC: Foreign objects in perl

[John Tobey]

[un-CC'ed to librarian - is this correct procedure?]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> 
> =head1 TITLE
> 
> A method of allowing foreign objects in perl

I like it.

> =head1 VERSION
> 
>   Maintainer: Dan Sugalski <[EMAIL PROTECTED]>
>   Date: August 02, 2000
>   Version: 1
>   Mailing List: perl6-internals
>   Number: XXX
> 
> =head1 ABSTRACT
> 
> Currently it's a darned pain to deal with objects created in other
> languages in perl. It would be very nice to be able to call a function
> written in some other language, take the returned object, and treat it
> as a perl object. (Which means making method calls and such on it)
> 
> =head1 DESCRIPTION
> 
> Everybody and their dog has written an object-oriented language at
> this point, and it'd be nice to be able to use objects created by
> those languages as if they were real perl objects. This would make
> interfacing perl and OO languages[1] easier, and make things much
> nicer for embedded and embedding code.
> 
> Since writing half a zillion perl2WhoKnowsWhat modules is going to be
> unweildy, not to mention different for each and every object type,
> perl punts. We require that the code we interface to defines a
> dispatch function for each object, and we then just call the dispatch
> function with enough information for it to Do The Right Thing.
> 
> >From the perl programmer's point of view, foreign objects and native
> objects look the same. Perl will define some generic conversion
> routines (to int, float, and string) which the foreign code is welcome
> to override if it wishes.
> 
> =head1 IMPLEMENTATION
> 
> A scalar is made an object via a call into the perl library. The
> scalar is marked as an object and stuck into a package. Attached to
> the scalar is a pointer to the native object, a pointer to a generic
> dispatch routine, and a pointer to the native destruction routine for
> the object.

Perhaps add a native mark routine, if we are really GC-agnostic.
Native serializer?  Equality tester?  Type id accessor?  What prompts
me to propose these is the struct that evolved in Perlmacs for mixing
Perl and ELisp objects:


struct Lisp_Foreign_Object_VTable
  {
Lisp_Object (*type_of)   P_ ((Lisp_Object self));
void(*destroy)   P_ ((Lisp_Object self));
void(*mark)  P_ ((Lisp_Object self));
Lisp_Object (*to_string) P_ ((Lisp_Object self));
int (*equal) P_ ((Lisp_Object self, Lisp_Object other, int depth));
Lisp_Object (*call) P_ ((Lisp_Object self, int nargs, Lisp_Object *args));
  };

-John

> Native code would create the object thusly:
> 
>   TheObj *foo;
>   SV *new_sv;
>   foo = new TheObj("A parameter");
>   sv = perl_new_sv();
>   perl_make_sv_object(sv, "Some::Package", foo, &dispatch_routine,
>   &destroy_routine);
> 
>   perl_return(perl_make_ref(sv));
> 
> When you make a method call in perl, like so:
> 
>   $native_obj->foo($bar, $baz);
> 
> perl calls the underlying dispatch routine to do the actual work. The
> dispatch routine has a function signature like so:
> 
>   int status = dispatch(void *native_obj, sv *perl_scalar, char *method_called,
>       int *num_args_in, perl_arg_stack *arg_stack,
>   int *num_args_out, perl_arg_stack *return_values);
> 
> 
> 
> =head1 REFERENCES
> 
> None I can think of
> 
> =head1 FOOTNOTES
> 
> [1] Like, say, C++ or Scheme
> 
> 

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Pickle now Helmi (was Re: GC)

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 04:38 PM 8/2/00 -0400, John Tobey wrote:
> >Can I safely assume that we will be counting references for garbage
> >collection?
> 
> No.
> 
> In fact, you can safely assume that GC will be handled by some non-refcount 
> method, and that GC and end-of-scope action will be decoupled. (Whether 
> that *happens* is a separate issue, but it was the plan)

No problem.  I'll have both an addref()/delref() pair and a
protect()/unprotect() pair and use all four functions where
appropriate.  One of the pairs will be no-ops, depending on which GC
we use.  Pickle may end up bringing mark+sweep to Perl 5 as a viable
option...

By the way, an announcement:  Pickle is now called HELMI, at the
suggestion of Brent Fulgham, who has volunteered to get in on
SourceForge.  Brent explains:

HELMI

(Helmi is Finnish for 'pearl').

We can then create some stupid acronym:

_H_euristic
_E_xtension
_L_anguage
and
_M_odular
_I_nterpreter

coupled with the more whimsical version:

_H_orrifyingly
_E_clectic
_L_anguage
of
    _M_ass
_I_nsanity

Thanks, Brent!

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

GC

[John Tobey]

Can I safely assume that we will be counting references for garbage
collection?

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: Feature request: Relocatable perl

[John Tobey]

> cc: [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED]
> Date: Wed, 02 Aug 2000 13:58:49 -0600
> From: Tom Christiansen <[EMAIL PROTECTED]>
> X-UIDL: ab65e2a94d9487f65465be74c21fe152
> 
> >> Insofar as you can pretend to make a good guess at that--perhaps.
> 
> >On Linux: /proc/self/exe
> 
> What good does that do you?  You can't go ../foo with that.  It doesn't
> tell you the real name, just the dev,ino.

Here:

$ ls -l /proc/self/exe
lrwx--1 jtobey   jtobey  0 Aug  2 16:06 /proc/self/exe -> /bin/ls
$ readlink /proc/self/exe
/bin/readlink
$ perl -le 'print readlink "/proc/self/exe"'
/home/usr/local/bin/perlmacs

>  And it's a nonportable answer
> that cannot be reproduced everywhere that Perl runs.

Touche.  I believe Windows' GetModuleFileName will work.  Any others?

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: Feature request: Relocatable perl

[John Tobey]

Tom!  Good to see you.

Tom Christiansen <[EMAIL PROTECTED]> wrote:
> >Yes, but that seems a bit more tractable - surely we could fiddle with
> >@INC based on the location of the perl interpreter?
> 
> Insofar as you can pretend to make a good guess at that--perhaps.

On Linux: /proc/self/exe

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Nick Ing-Simmons <[EMAIL PROTECTED]> wrote:
> John Tobey <[EMAIL PROTECTED]> writes:
> >> I don't think so - it is a question which way we code the source:
> >> 
> >> A. Use 'inline' every where and trust compiler not to do what we told it 
> >>if it knows better.
> >> B. No inline hints in the source and trust the compiler to be able to 
> >>do the right thing when prodded with -O9 or whatever.
> >> C. Make "informed" guesses at to which calls should be inlined.
> >> 
> >> My view is that (B) is the way to go, aiming for (C) eventually, because 
> >> (A) gives worst-case cache purging.
> >
> >There doesn't have to be an 'eventually'.  We can have (C) now.
> 
> Then you know perl's execution profile a lot better than I do ;-)
> Pop Quiz:

Please read everything before replying to anything.  :-)

> >Moving from a first-guess (C) to an optimal (C) (where we make
> >reasonable hints all the time, no doubt with the aid of some Configure
> >tests or machine-dependent conditionals) can be an ongoing pursuit.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

performance of inlined code (was Re: inline mania)

[John Tobey]

Nick Ing-Simmons <[EMAIL PROTECTED]> wrote:
> John Tobey <[EMAIL PROTECTED]> writes:
> >> That sounds like a pre-judgement that inline form is prefered ;-)
> >> But actually most places (I looked when doing TIEARAY) perl itself calls
> >> nether it uses AvFILL() macro instead.
> >
> >As I understand myself, AvFILL == i_av_fill, so thank you for
> >confirming my pre-judgment.  :-)
> 
> No - one more time - your prejudgement may match perl5 - my point is
> PERL5 GOT IT WRONG (I think, and other hardware aware types Alan, Dan, ...
> seem to agree).

Ah, okay.  You mean perl5 got it wrong.  I'm no CPU architect, as you
may have noticed.  I'm merely reporting what I've observed in perl5.

Thanks for clarifying this.

-John

> It may have been right for a 3MHz 68000 based Sun3 but it isn't for 
> UltraSPARC, Pentium-III, IA-64, ...
> 
> -- 
> Nick Ing-Simmons
> 
> 

XS shortcomings (was Re: inline mania)

[John Tobey]

Nick Ing-Simmons <[EMAIL PROTECTED]> wrote:
> Tim Bunce <[EMAIL PROTECTED]> writes:
> >
> >Perl5 XS/GUTS is the way it is largely because it was designed with
> >performance in mind.  
> 
> Hmm, maybe in some areas but not others:
>  On the one hand the whole PUTBACK/SPAGAIN mess is by overly clever 
>  attempt to optimize access to "global" variables which compiler 
>  has a better chance of getting right by itself.

Easily remedied:

#undef dSP
#define dSP dNOOP
#undef PUTBACK
#define PUTBACK NOOP
#undef SPAGAIN
#define SPAGAIN NOOP
#define sp PL_stack_sp

>  While on the other hand XS is forced to stuff everything on the stack.
>  My main complaint is that XS code is too unlike pp_xxx() and you 
>  cannot easily call into pp_xxx().   

Yep, everything except the dPOPs, dATARGET, etc. stuff should be
factored out and called as a separate function, similar to the way
pp_join uses do_join:

PP(pp_join)
{
djSP; dMARK; dTARGET;
MARK++;
do_join(TARG, *MARK, MARK, SP);
SP = MARK;
SETs(TARG);
RETURN;
}

> >I think Larry did a very good job. My only big
> >worry for perl6 is that, without great care, we'll loose that
> >performance edge in the name of a 'clean API'.

Please everyone accept that we will provide *both* a source *and* a
binary API, so ideally the choice between total chastity and total
incest happens in a single #define.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Nick Ing-Simmons <[EMAIL PROTECTED]> wrote:
> Brent Fulgham <[EMAIL PROTECTED]> writes:
> >The 'inline' keyword is just a hint to the compiler.  If optimization
> >is turned off, no inlining is done.  If optimization is on, the
> >compiler may or may not decide to inline.  Performance on different
> >compilers will vary.  
> >
> >To repeat:  Even if I say "inline" on everything, the compiler is
> >free to disregard that if its optimization routines decide not to.
> >(Also, if I fail to say "inline" on something, the compiler may
> >decide to inline if optimization is active).
> >
> >So aren't we all saying the same thing?
> 
> I don't think so - it is a question which way we code the source:
> 
> A. Use 'inline' every where and trust compiler not to do what we told it 
>if it knows better.
> B. No inline hints in the source and trust the compiler to be able to 
>do the right thing when prodded with -O9 or whatever.
> C. Make "informed" guesses at to which calls should be inlined.
> 
> My view is that (B) is the way to go, aiming for (C) eventually, because 
> (A) gives worst-case cache purging.

There doesn't have to be an 'eventually'.  We can have (C) now.
Having both forms available (i.e., the suggestion that started this
thread) lets us 'help' the compiler, which, as Brent points out, is
free to ignore our hint under normal optimization flags.

Moving from a first-guess (C) to an optimal (C) (where we make
reasonable hints all the time, no doubt with the aid of some Configure
tests or machine-dependent conditionals) can be an ongoing pursuit.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Nick Ing-Simmons <[EMAIL PROTECTED]> wrote:
> John Tobey <[EMAIL PROTECTED]> writes:
> >Dan Sugalski <[EMAIL PROTECTED]> wrote:
> >> Bad assumption. How often is av_fill called?
> >
> >Only once in av_fill.c (generated by allfuncs.pl).  In most other
> >places, it's called as i_av_fill().
> 
> That sounds like a pre-judgement that inline form is prefered ;-)
> But actually most places (I looked when doing TIEARAY) perl itself calls
> nether it uses AvFILL() macro instead.

As I understand myself, AvFILL == i_av_fill, so thank you for
confirming my pre-judgment.  :-)

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: Feature request: Relocatable perl

[John Tobey]

[EMAIL PROTECTED] wrote:
> Graham Barr wrote:
> 
> > Well I think it is worth an RFC (not that I have time to write one just now)
> > If only to suggest some sample implementations. Do other languages do it ?
> > If so how ?
> 
> Ok, Ok, I'll do a RFC then...   :-)
> 
> Actually the only thing affected by LD_LIBRARY_PATH is libperl.so, as
> modules are loaded in by dlopen() or equivalent, so the full path in
> that case is specified in the call to dlopen().
> 
> Solaris 7 onwards has a nice linker feature called $ORIGIN.  This allows
> the location of run-time linker dependencies to be specified relative to
> the location of the executable.

Both Windows and Linux can swing this, too.  I believe linux (GNU
Libc, to be precise) uses $ORIGIN in a Solaris-compatible way.

I don't mean to imply, however, the binediting perl has no place.

-John

>  This is probably best illustrated with
> a simple example:
> 
> $ pwd
> /tmp/app
> $ cat bin/main.c
> extern void hi();
> int main() { hi(); }
> $ cat lib/hi.c
> #include 
> void hi() { printf("hi!\n"); }
> $ cc -G -K pic -o lib/libhi.so lib/hi.c   
> $ cc -Llib -R '$ORIGIN/../lib' bin/main.c -o bin/main -lhi
> $ ldd bin/main
> libhi.so =>  /tmp/app/lib/libhi.so
> libc.so.1 => /usr/lib/libc.so.1
> libdl.so.1 =>/usr/lib/libdl.so.1
> /usr/platform/SUNW,Ultra-4/lib/libc_psr.so.1
> $ bin/main
> hi!
> $ cd ..
> $ mv app somewhere_else
> $ ldd somewhere_else/bin/main
> libhi.so =>  /tmp/somewhere_else/lib/libhi.so
> libc.so.1 => /usr/lib/libc.so.1
> libdl.so.1 =>/usr/lib/libdl.so.1
> /usr/platform/SUNW,Ultra-4/lib/libc_psr.so.1
> $ somewhere_else/bin/main
> hi!
> 
> Another option for ELF based platforms that lack $ORIGIN (i.e.
> everything other than Solaris AFAIK) would be to use the (standard?)
> libelf library to write a little application that tweaked the RPATH in
> the executable appropriately when perl is installed somewhere else other
> than the default location.  I think this would be far preferable to the
> currently used horrible hacks involving either LD_LIBRARY_PATH or
> greping and patching the executable.
> 
> -- 
> Alan Burlison
> 
> 

Re: Forging ahead alone (or "I can rewrite perl myself!") (was Re: inline mania)

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> I'll point out here that I, for one, don't mind people deciding to head out 
> and do a port/reimplement/rewrite on their own. I won't promise that your 
> work (whoever you are in that sentence) will go into the 'official' perl 

Thanks, that's all I ask.  I hope anyone with the time and ability
tries to do their own port/reimplement/rewrite as a homework exercise,
just as Chip did and now I'm trying to do.

> implementation, though I won't nick code from anyone without permission. 
> (Ideas, yes, but code, no)

In my case, any code that I write and mention here will carry a
license that permits its being nicked for any official perl release.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Tim Bunce <[EMAIL PROTECTED]> wrote:
> So please, follow Chips wise lead, don't call your work Perl 6.

It's called Pickle.

"Andy Wardley" <[EMAIL PROTECTED]> wrote:
> Indeed.  XS is hard, fast, dirty and ugly (in a sickly, beautiful
> kinda way), but there's nothing to stop you from wrapping it all up
> into a less efficent, but prettier API (with the possible exception
> of reference counting).

There is no reason for it to be any less efficient (other than
compilers' botching of `inline').  I favor leaving efficiency as a
user option.  In return for efficiency, we offer binary independence
of interpreters.

Pickle provides (or will provide) three modes of compiling:

Default -  is not included - total bincompat
Refcount Cheat - the only bit structure knowledge is sv_refcount
Gung Ho - all Default API functions are inline to the level of
Perl 5's API

You can select any of them with a single conditional.  I believe Perl
6 (and v5.8) should support three interfaces along these lines,
whether or not any Pickle code is involved.

-John

Re: inline mania

[John Tobey]

Tim Bunce <[EMAIL PROTECTED]> wrote:
> There may be scope for having two levels of API. One high level clean
> simple etc and the other a lower-level more performance oriented API.
> The former could be implemented in terms of the latter.

This is essentially what I meant by:

inline i_foo() { BLA; }
...
foo() { i_foo(); }

-John

Re: inline mania

[John Tobey]

Sam Tregar <[EMAIL PROTECTED]> wrote:
> On Tue, 1 Aug 2000, John Tobey wrote:
> 
> > The people here are rightly skeptical about the effectiveness of using
> > the 5.6 code base as a starting point for v6, but I have a pretty
> > clear vision of how to do it, and I am committed to giving it a try,
> > even if no one else will.  In fact, I'll give you all a tentative
> > schedule:
> 
> Wait, you're going to develop Perl 6 ALONE?  Wasn't this going to be "the
> community's rewrite of Perl"?  Shouldn't you be trying to rally support
> for your vision before issuing schedules?

Not really.  I'm going to START developing MY IDEA of how Perl 6 might
work.  Very likely, I'll stop or merge with someone else's ideas along
the way.

The schedule is just to get something down, to start fleshing out my
idea and maybe provoke other people into starting on their own
visions, as Simon Cozens has (though he seems to deny it) done with
GLib and Sapphire.

> I'm not trying to knock you - I'm not at all against hearing you plans and
> possibly helping out.  This just seems like a pretty strange way to
> approach a community effort.
> 
> > 15 August 2000   - detailed draft spec to perl6-internals.
> > 31 August 2000   - revised spec after discussion.
> 
> What?  You're expecting all the various perl6-* lists to come up with
> final RFCs be the end of the month?  And you're expecting to have Larry's
> final plans by then?

Well, I'll read RFC's as they appear and adjust to them as they are
approved, but the stuff I want to do now does not need to wait.

> This sounds hopeful, but mostly unfounded.  Without starting with
> threading and Unicode as primary features you're going to be fighting an
> uphill battle ala Perl 5.

This I hope to address in my first architecture document.

Thanks for the comments.
-John

Re: inline mania

[John Tobey]

[EMAIL PROTECTED] wrote:
> John Tobey wrote:
> 
> > > Why is he bothering?  A year to produce a prototype doesn't seem like a
> > > useful way to expend effort on something that isn't actually perl6.
> > 
> > It is actually perl6 if/when it's finished.
> 
> Right, so it isn't a community effort then, as you intend doing it all
> yourself.

I always welcome patches, suggestions, and constructive criticism.
And if somebody else wants the "maintainer" role, I'll be happy to let
them do that work for me.  :-)

> > > I'm not saying it was.  However I don't see how the proposal would aid
> > > the migration - after all what he is writing will be neither perl5 nor
> > > perl6.
> > 
> > I am not "writing".  I am "transforming".
> 
> Ok, so neither is it a rewrite.  I suspect it won't be called perl6 in
> any case.

Well, if not, I'll have the satisfaction of having tried.

Cheers
-John

Re: Nick's performance theory - was "inline mania"

[John Tobey]

Nick Ing-Simmons <[EMAIL PROTECTED]> wrote:
> John Tobey <[EMAIL PROTECTED]> writes:
> >Nick Ing-Simmons <[EMAIL PROTECTED]> wrote:
> >> But is usually much easier add entropy - so start with its the same 
> >> function - call it, and let compiler decide which ones to expand.
> >
> >You'll get no argument on that point.  Please stop suggesting that I
> >want to take the power of decision away from programmers *OR*
> >compilers.
> 
> Fine. All _I_ am saying is if we write it as a normal function in 'PI'
> we can easily spew either form. We don't need to decide now.

Excellent.

> The 'PI' source should NOT specify whether to define nor whether to call
> the inline form vs the real function form.

My fault for not thinking in the PI mindset.

> >> We can unlink the .c files once we have compiled them ;-)
> >
> >Nope.  Messes up source debuggers.
> 
> 
> A. I was jesting.
> B. There will of course have been lots of 
> #line "pp_hot.pi" 1417
> directives so debugger refers you to what programmer wrote (this is _vital_ or 
> people hit their favourite hot-key and fixup the generated .c file not 
> the source!).

Of course!  My first and proudest contribution to Perl was a fixup of
xsubpp's #line directive emission.  :-)

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> Bad assumption. How often is av_fill called?

Only once in av_fill.c (generated by allfuncs.pl).  In most other
places, it's called as i_av_fill().

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 05:34 PM 8/1/00 -0400, John Tobey wrote:
> >Okay.  For starters, assume that every inline function is called in
> >exactly one place in the translation unit that defines its non-inline
> >counterpart.  That one place being, of course, i_foo's foo.  This is a
> >natural result of a clean, PI-like-generated source.
> 
> Bad assumption. How often is av_fill called?

By "assume" I mean "ensure" here.  As in, this is how we build our
library.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 05:55 PM 8/1/00 -0400, John Tobey wrote:
> >Dan, you are completely missing my point.  Okay, fine, non-inline may
> >be a performance win in some cases.  Inline may be a win in others.  I
> >am not proposing we mandate inlining in any case, I am proposing the
> >exact opposite: that we let the caller decide in every case.
> 
> Having thought about it a bunch more (because of this) I'm proposing we let 
> the compiler decide. The caller doesn't know enough to make that decision. 

Read carefully.  I said we *let* the caller decide, not *make* the
caller decide.  What, specifically, disturbs you about my proposal?

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Alan Burlison <[EMAIL PROTECTED]> wrote:
> Brent Fulgham wrote:
> 
> > > I think there is an undiscussed assumption about the implementation
> > > language in there somewhere...
> > 
> > I think you may have missed the context of the message.  John was talking
> > about creating his Alpha using various existing projects that had already
> > been done in C++.
> 
> Why is he bothering?  A year to produce a prototype doesn't seem like a
> useful way to expend effort on something that isn't actually perl6.

It is actually perl6 if/when it's finished.

> > > We've been down that path already - Topaz.  With all due respect this is
> > > supposed to be a community rewrite.  Your proposal doesn't seem to be
> > > along those lines.
> > 
> > With all due respect, I think you may be taking this out of context.  I
> > don't believe John's intent was to hijack the process.  He was outling
> > a theoretical schedule that could be used to provide a working
> > Perl5 -> Perl6 migration path.
> 
> I'm not saying it was.  However I don't see how the proposal would aid
> the migration - after all what he is writing will be neither perl5 nor
> perl6.

I am not "writing".  I am "transforming".

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: Nick's performance theory - was "inline mania"

[John Tobey]

Nick Ing-Simmons <[EMAIL PROTECTED]> wrote:
> But is usually much easier add entropy - so start with its the same 
> function - call it, and let compiler decide which ones to expand.

You'll get no argument on that point.  Please stop suggesting that I
want to take the power of decision away from programmers *OR*
compilers.

> >If someone else wants to prove this, great.  I just don't think it's
> >that much trouble.  (mostly psychological - what will people think if
> >they see that all our code is in headers and all our C files are
> >autogenerated?)
> 
> We can unlink the .c files once we have compiled them ;-)

Nope.  Messes up source debuggers.

Re: inline mania

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 09:43 PM 8/1/00 +, Nick Ing-Simmons wrote:
> >Nor am I sure that the XS API is not one of the things I would not
> >have near top of the list for "internals cleanup":
> >
> >PUSHMARK;
> >XPUSHs(sv_2mortal(newSVpv('DeepThought'));
> >XPUSHs(sv_2mortal(newSViv(42));
> >PUTBACK;
> >count = perl_call_method("Consider",G_EVAL);
> >SPAGAIN;
> >if (count >= 1) {
> >  answer = SvIV(POPs));
> >}
> >
> >You _want_ to keep that???
> 
> I want to print it out, take it out to the parking lot, drive a stake 
> through it, light it on fire, bury the ashes print side down into a silver 
> box inside an iron box under a crossroad sprinkled with garlic powder. Then 
> I want to do something drastic to it...

I nominate Dan to rewrite DBI.xs...

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Brent Fulgham <[EMAIL PROTECTED]> wrote:
> > Alan Burlison wrote:
> >
> > John Tobey wrote:
> > 
> > > 1 November 2000  - Perl6 alpha in C++ that uses classes derived
> > >from PerlInterpreter and SV 
> > everywhere in place
> > >of these types, a la Pickle, but with inline
> > >methods that use Perl 5's internal API.
> > 
> > I think there is an undiscussed assumption about the implementation
> > language in there somewhere...
> > 
> 
> I think you may have missed the context of the message.  John was talking
> about creating his Alpha using various existing projects that had already
> been done in C++.

Kind of.  The only "existing project" in C++ is Pickle, but I plan to
make fairly light use of C++, probably without any mentions of
`virtual'.  This is required for bincompat, which is required as
intermediate versions are based partly on v5.6 code and partly on new
code.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Nick Ing-Simmons <[EMAIL PROTECTED]> wrote:
> By all means have a go at Topaz-II 
> 
> But it won't be 'Perl 6' unless it implements what perl-language 
> decides that _is_.

Agreed.  It won't even remotely resemble 'Perl 6' (as opposed to Perl
5) until it's in a position to start changing the lexer, at least
several months from now.

> Hmm - personally I cannot imagine how you can maintain XS / API compatibility
> and improve things ...

Well, that's what I hope to explain in my August 15 spec.  :-)

> Nor am I sure that the XS API is not one of the things I would not 
> have near top of the list for "internals cleanup":
> 
> PUSHMARK;
> XPUSHs(sv_2mortal(newSVpv('DeepThought'));
> XPUSHs(sv_2mortal(newSViv(42));
> PUTBACK;
> count = perl_call_method("Consider",G_EVAL);
> SPAGAIN;
> if (count >= 1) {
>  answer = SvIV(POPs));
> }
> 
> You _want_ to keep that???

Off to the side, out of sight of new code, yes.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Alan Burlison <[EMAIL PROTECTED]> wrote:
> John Tobey wrote:
> 
> > 1 November 2000  - Perl6 alpha in C++ that uses classes derived
> >from PerlInterpreter and SV everywhere in place
> >of these types, a la Pickle, but with inline
> >methods that use Perl 5's internal API.
> 
> I think there is an undiscussed assumption about the implementation
> language in there somewhere...

Yes.  "C or C++".  Notice that I said a semi-goal is to make the end
product rewritable in plain C.  I am willing to upgrade this to a
full-fledged goal if it pleases the masses.

> We've been down that path already - Topaz.  With all due respect this is
> supposed to be a community rewrite.  Your proposal doesn't seem to be
> along those lines.

Well, since I'm not asking anything (much) of you, I guess I don't
have to answer this.  But somebody has to do the work, and preferably
somebody in deep hack mode with nothing else in his/her mind to crowd
out the Perl.

I've got all my code in a publicly accessible CVS server, and I'd be
happy to move it to someone else's publicly accessible CVS server (or
Perforce or Perverse, if it's all it's cracked up to be).

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> > > >Non-inline functions have their place in reducing code size
> > > >and easing debugging.  I just want an i_foo for every foo that callers
> > > >will have the option of using.
> > >
> > > Before we make any promises to do all that extra work can we
> > > measure (for various architectures) the cost of a real call vs inline.
> > >
> > > I want proof that inline makes X% difference.
> >
> >I'm not going to prove that.  A normal C function call involves
> >several instructions and a jump most likely across page boundaries.
> 
> But if that function is already in cache from another use, you win.
> 
> Assume, for a minute, you've got a 700MHz system with a 100MHz 128-bit data 
> bus. If your code is inlined and works out to 256 bytes, that's 16 fetches 
> on the main bus. That costs 112 cycles. On the other hand, if your 

Dan, you are completely missing my point.  Okay, fine, non-inline may
be a performance win in some cases.  Inline may be a win in others.  I
am not proposing we mandate inlining in any case, I am proposing the
exact opposite: that we let the caller decide in every case.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 04:55 PM 8/1/00 -0400, John Tobey wrote:
> >Well, I am going to assume you are wrong and the above two foo()
> >implementations will produce exactly the same code.
> 
> Your assumption is incorrect for good compilers. (Whether gcc counts as 
> good for a particular platform is a place I'm Not Going right now

Agreed.  :-)

> ) Compaq C 
> will do it differently depending on the number of times that the inlined 
> function is used.

Okay.  For starters, assume that every inline function is called in
exactly one place in the translation unit that defines its non-inline
counterpart.  That one place being, of course, i_foo's foo.  This is a
natural result of a clean, PI-like-generated source.

Other files can choose between the foo and i_foo versions, but foo's
compiled form will not be changed, of course, unless the linker gets
in on the act.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Sam Tregar <[EMAIL PROTECTED]> wrote:
> On Tue, 1 Aug 2000, John Tobey wrote:
> 
> > I'm thinking of a rewrite in the sense that (by my impression) BSD was
> > a rewrite of Unix.  Refactoring, shifting stuff around until it's easy
> > to get a handle on things, and replace it all a little at a time.  So,
> > yes, these problems can be solved IMHO, after "enough" of Perl 5 has
> > been rewritten.
> 
> Is that Perl 6 you're talking about or Perl 5.8?  I mean that.  Perl 5
> will continue its development while we're doing the rewrite. Incremental
> changes would be best done in the 5.7 series, no?

Perl 6.  5.8 will contain incremental improvements and added features,
but 6.0 (what I'll call "Morph 6.0" as opposed to "Scratch 6.0") would
be much more aggressively changed.  Backwards compatibility would be
salable, source files rearranged, and the guts scarcely recognizable
in the end, but as a result of evolution, not revolution.

The people here are rightly skeptical about the effectiveness of using
the 5.6 code base as a starting point for v6, but I have a pretty
clear vision of how to do it, and I am committed to giving it a try,
even if no one else will.  In fact, I'll give you all a tentative
schedule:

15 August 2000   - detailed draft spec to perl6-internals.
31 August 2000   - revised spec after discussion.
1 November 2000  - Perl6 alpha in C++ that uses classes derived
   from PerlInterpreter and SV everywhere in place
   of these types, a la Pickle, but with inline
   methods that use Perl 5's internal API.
1 June 2001  - Perl v6.-1.0 (eek!) that passes all tests
   (troublesome tests will have been removed or
   changed) and has in place the necessary
   abstractions for unplugging and plugging core
   component implementations.  Probably a GLib
   plugin proof-of-concept.

At least once a month, though not at regular intervals, a report of
the test suite status will be posted on the Web, including tests
failed and some discussion of why and what is to be done.  A lengthy
discussion will be required if more than 20% of test scripts are
removed or fail for any given month.

This schedule will be accelerated about 2x if somebody hires me to
work full time on it.  (AS: My requirements are modest! :-)

I will start by using C++ where applicable for ease of development,
but I will try to keep open the possibility of switching the core
implementation back to C once it's past alpha.

Unicode and threading would become integrable only after a lot of
morphing (refactoring).  The morphing would probably destroy any
traces of v5 unicode support (since well under 20% of test scripts
will notice), and of course 5005threads will be the first to go.  With
any luck, a compatible, well-integrated replacement will eventually
take its place.

What drives this vision, you might ask?  Compatibility.  XS and API
source compatibility with Perl 5 will be much stronger this way than
with Scratch 6.0.  Binary compatibility layers such as Pickle will be
more practical, and unexpected quirky differences will be minimal.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Nick Ing-Simmons <[EMAIL PROTECTED]> wrote:
> John Tobey <[EMAIL PROTECTED]> writes:
> >Is foo() compiled any differently in
> >
> >inline i_foo() { BLA; BLA; BLA; }
> >foo() { i_foo(); }
> >
> >versus
> >
> >foo() { BLA; BLA; BLA; }
> 
> Maybe not for void functions with no args, tail-called and with 
> no prefix, but in more typically cases yes it can be different
> the "function-ness" of i_foo applies constaints on where args
> and "result" are which optimizer _may_ not be able to unravel.

May not be able because of what the Standard says, or because of
suboptimal optimization?

> >?  I am not talking about forcing inlined functions down anyone's
> >throat.  
> 
> I believe that if function is defined before use "modern" C compilers
> (e.g. gcc) can inline functions not declared such.

GCC won't unless you go -O3 or above.  This is why many people (me
included) stop at -O2 for most programs.

> >Non-inline functions have their place in reducing code size
> >and easing debugging.  I just want an i_foo for every foo that callers
> >will have the option of using.
> 
> Before we make any promises to do all that extra work can we 
> measure (for various architectures) the cost of a real call vs inline.
> 
> I want proof that inline makes X% difference.

I'm not going to prove that.  A normal C function call involves
several instructions and a jump most likely across page boundaries.
If someone else wants to prove this, great.  I just don't think it's
that much trouble.  (mostly psychological - what will people think if
they see that all our code is in headers and all our C files are
autogenerated?)

I will, however, take your suggestion about comparing gcc -S output
for examples like mine above.

> If that proof is there and inline is a "significant win" then 
> we teach "PI" or whatever ends up being used instead of #define
> to emit the above when "source" defines foo().

Yes.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 04:37 PM 8/1/00 -0400, John Tobey wrote:
> >Is foo() compiled any differently in
> >
> > inline i_foo() { BLA; BLA; BLA; }
> > foo() { i_foo(); }
> >
> >versus
> >
> > foo() { BLA; BLA; BLA; }
> >
> >?
> 
> Probably. On many newer chips, the first form will probably get you a 
> function called i_foo and foo will call it, rather than actually inlining 
> the code. (Depends on how often i_foo is called) Function calls are 
> reasonably cheap, and if you have one copy of the function instead of many, 
> it may well be in the cache already, where it wouldn't be in the inlined case.
> 
> Given that some of the faster machines take a 10x speed hit getting to main 
> memory, that can make a *big* difference.

Well, I am going to assume you are wrong and the above two foo()
implementations will produce exactly the same code.  I think GCC lets
you control whether i_foo becomes a bona fide symbol by letting you
declare it `extern inline'.  In any case, I do *not* anticipate many
direct calls to i_foo() in any given translation unit.

> (FWIW, I was thinking of the difference between #define-ing a function and 
> just declaring it--I'd forgotten about the inline keyword)

Please, let's assume `inline'.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
>  (FWIW, it seems on many of the modern 
> processors that inlining code decreases your performance, so I think 
> deciding on stuff like that is rather premature)

Is foo() compiled any differently in

inline i_foo() { BLA; BLA; BLA; }
foo() { i_foo(); }

versus

foo() { BLA; BLA; BLA; }

?  I am not talking about forcing inlined functions down anyone's
throat.  Non-inline functions have their place in reducing code size
and easing debugging.  I just want an i_foo for every foo that callers
will have the option of using.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: [External API's: XS, Pickle, Win32::API, FFI, C::DynaLib etc.

[John Tobey]

Dan Sugalski <[EMAIL PROTECTED]> wrote:
> At 11:58 AM 8/1/00 -0500, Garrett Goebel wrote:
> >From: John Tobey [mailto:[EMAIL PROTECTED]]
> > >
> > > Since this issue mainly affects Windows users (I assume),
> >
> >Actually I've run across a couple people wanting this on unices.
> 
> I want it on VMS, too.

Tell me whether C::DynaLib[1] passes the tests.  It worked on
Alpha/DU.  No VAX support, though.

> Creating an interface to a generic library without wrapper code
> would be a reasonably nifty thing. It'd take care of a good chunk of
> what Xs is used for. (We could reduce the average library interface
> module to a .PM file and a .DYNA_SPEC one, or something similar)

Problem: how to parse header files for type sizes, etc.

> >I guess I don't understand the pro's/con's of XS glue code
> >vs. something like FFI, Win32::API, or C::DynaLib. Can anyone
> >elaborate on this?
> 
> I'd like to see this info as well. I'm contemplating something
> similar for the embedding interface for perl, so maybe we can
> dovetail the two together or something.

Well, since I wrote two[1][2] of the interfaces mentioned in the
subject line, I'll do the elaborating...

> >Whatever eventually replaces XS, I certainly hope that coding to it
> >feels more like Perl working with C, than mystic C macros working
> >with Perl.
> 
> It'll probably be more like C calling into a nice, well-behaved
> library with a well-defined interface that hands around a lot of
> magic cookies.

Yes.  One thing lacking in Perl 5 is an API that exposes the semantics
of Perl, the language, while meticulously hiding everything else.
This is what Pickle[2] intends to provide for Perl 5 and all future
Perls.  Pickle is C++, but we should also have a C interface.  Both
interfaces should come in bincompat and sourcecompat (i.e.,
inline/macroized) flavors.

(In my wilder moments, I foresee a grander role for Pickle in Perl 6's
development, but that is for another mail.)

C::DynaLib[1] is a Perl interface to C compiled code that works (more
or less) on IA32, SPARC, and Alpha machines.  It uses DynaLoader to
load a library and lets you construct coderefs by giving a symbol name
and C codes for the function argument and return types.  It also
has rudimentary callback support.

C::DynaLib is kind of lightweight and easy to use (when it works), but
I consider it a hack, do not use it myself, and would be happy to give
it to a new maintainer.  I was inspired to write it, not out of need,
but because I had used a similar feature in Visual Basic and wanted
motivation for learning Perl's internals.

FFI[3] is a newer CPAN module by Paul Moore based on GNU Ffcall
("foreign function call").  Ffcall is much more ambitious than DynaLib
in the number of architectures supported and the completeness and
robustness of the implementation.  Additionally, it supports
generating first-class closures as C function pointers.  GNU Guile
(Scheme) has a good set of Ffcall bindings, I believe.

Ffcall is licenced under the self-propagating GPL.  I don't know who
holds the copyright or whether they would consider making exceptions.

XS[4] is Perl's standard interface for calling external C code.  Both
C::DynaLib and FFI are implemented in XS and augment it with support
for passing various types of arguments to C functions whose prototypes
are not known until run time (i.e., script-writing time).

If you know the function you want to call and have a working XS and C
compiler (typically because you built perl from source), then you can
write an XS module and use that instead of FFI or the like.  This is
generally more robust and preferred, but it takes a bigger time
investment, especially with respect to learning the XS language and
the Perl API.  Pickle aims to reduce this requirement substantially
for C++ programmers.

As for Win32::API, I don't know much about it.  I've never used it,
but I think it is a set of XS modules that let Perl use the most
popular Windows API functions.  The problem with this is that there
are just too many functions in too many Windows DLLs for XS coders to
keep up with, and end users who don't have compilers would like to
call the functions without waiting.

Hope this helps.
-John

[1] http://search.cpan.org/search?mode=module&query=C%3A%3ADynaLib
[2] http://search.cpan.org/search?dist=pickle
[3] http://search.cpan.org/search?mode=module&query=FFI
[4] http://search.cpan.org/search?mode=doc&query=perlxs

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: RFC: On-the-fly tainting via $^T

[John Tobey]

Simon Cozens <[EMAIL PROTECTED]> wrote:
> On Tue, Aug 01, 2000 at 01:43:05PM +0100, Graham Barr wrote:
> > Let me just say that Larry has said in the past that untainting was
> > deliberatly left difficult to do, on the basis that something which
> > can have serious effect (ie security) should not be easy to do.
> > 
> > But then I suppose all previous decisions are up for re-deciding
> 
> Yes, they are. If we're going to make it trivially easy to untaint,
> should we bother having tainting at all? :(

Tainting has potential uses as data-tracking mechanism aside from
security.  If the keyword 'untaint' had to appear, it would be easier
to find security issues than when m/(.*)/ is used.

Uh-oh, now we're getting back into perl6-language territory...
attempting to CC.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: External API's: XS, Pickle, Win32::API, FFI, C::DynaLib etc.

[John Tobey]

> [Reply redirected to perl6-internals, as requested by Dan Sugalski]
> 
> At 13:20 -0500 2000-07-31, Garrett Goebel wrote:
> >- Pickle looks like a better XS. But I don't know much about it
> 
> What Pickle would that be?  Python has something with the right name, 

Gah!!  That shows my ignorance of Python.  I, too, did a (fairly
cursory) web search before naming Pickle.  It's a better XS and Embed
for C++; see http://search.cpan.org/search?dist=pickle and my recent
postings to perl6-internals.

-John

> but "[t]he pickle module implements a basic but powerful algorithm 
> for ``pickling'' (a.k.a. serializing, marshalling or flattening) 
> nearly arbitrary Python objects. This is the act of converting 
> objects to a stream of bytes (and back: ``unpickling''). "  That 
> (from  
> ) doesn't seem to me to describe an external API.  If you mean 
> something else, can you supply a reference as a (fairly cursory) 
> search of the web shows me nothing.
> -- 
> Dominic Dunlop
> 
> 

Re: inline mania

[John Tobey]

OK, this is the last I'll say on it before laying *myself* to rest.
:-)

Simon Cozens <[EMAIL PROTECTED]> wrote:
> On Tue, Aug 01, 2000 at 02:31:15AM -0400, Sam Tregar wrote:
> > Note that we don't have to take that "chance," but I don't think we should
> > turn it down lightly.  I suppose a reasonable question is: can we achieve
> > our goals without a rewrite?  Can we succeed in integrating threading and
> > Unicode where the Perl5 developers have failed, without rewriting the
> > internals?  I'm not qualified to say no, but I'd like to hear compelling
> > arguments before I'd believe a yes!
> 
> I don't think I'm qualified to say no, but if I was, I'd say no. Too much
> in perl 5 has been bolted on as afterthoughts. Things like threading really
> need to be built in from the beginning. Unicode is currently a devious hack
> which is steadily working, but to get it *right* would mean ripping out
> reasonably big chunks of code. And there isn't very much room to maneuver
> in the internals right now. 

Ripping chunks out is acceptable as a means to an end.  The only
requirement is that 80% of the tests pass at least once a month.

I'm taking a top-down approach, starting with a library that just
thinly masks the Perl 5 implementation (i.e., perl.h) while exposing
the core semantics.  Some semantics I don't choose to support because
they are hairy, non-useful, or generally regarded as bugs.  As I start
pilfering code from Perl's *.c files for my own *.hh's, I'll trim out
enough to break 20% of the test suite if it'll reduce the complexity.
Then later we'll fix (most of) the 20%.

-John

> It's like a delicately balanced stack of cards: changes are very rarely
> atomic, and usually end up with you burbling around in toke.c (this is not
> fun) or in the macro hell that is deep SV and GV manipulation. If you want a
> fun afternoon, look at how UTF8 stashes are implemented. And then there's the
> regular expression engine.
> 
> Time to lay this to rest, I think.
> 
> -- 
> I think...  I think it's in my basement... Let me go upstairs and check.
>   -- Escher
> 
> 

Re: inline mania

[John Tobey]

> > Umm, well, not necessarily.  Not if what I'm doing becomes what we're
> > doing. 
> 
> Do tell!  What have you been doing?

Well, I don't want everybody looking at it in its current (TEMPORARY!)
shambles, but I posted download instructions to this list in a reply
to Simon.

> I'm not here to try to tell everyone we're doing a rewrite - that's only
> what I was told coming in the door.  From the Perl 6 press release:
> 
>The next version of Perl is a chance for the language
>developers to both rewrite the internals and externals of
>Perl based on their experience from developing Perl 5, and
>Chip Salzenbergs work with Topaz. 
> 
> Note that we don't have to take that "chance," but I don't think we should
> turn it down lightly.  I suppose a reasonable question is: can we achieve
> our goals without a rewrite?  Can we succeed in integrating threading and
> Unicode where the Perl5 developers have failed, without rewriting the
> internals?  I'm not qualified to say no, but I'd like to hear compelling
> arguments before I'd believe a yes!

I'm thinking of a rewrite in the sense that (by my impression) BSD was
a rewrite of Unix.  Refactoring, shifting stuff around until it's easy
to get a handle on things, and replace it all a little at a time.  So,
yes, these problems can be solved IMHO, after "enough" of Perl 5 has
been rewritten.

Good night.
-John

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Sam Tregar <[EMAIL PROTECTED]> wrote:
> We need to start with the basics and we need to keep things simple and
> understandable.  I get the sense that the perl dev community has more than
> its share of efficiency experts.  That's a good thing but there's a proper
> time to optimize and this is not it.
> 
> We're doing a rewrite, aren't we?

Umm, well, not necessarily.  Not if what I'm doing becomes what we're
doing.  See http://joel.editthispage.com/stories/storyReader$47 on the
evils of total rewrites.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

Re: inline mania

[John Tobey]

Simon Cozens <[EMAIL PROTECTED]> wrote:
> Counter-counterproposition: Look at the way Sapphire does it. I'm really
> proud of that. It's automagically switchable from inlines to ordinary
> functions to macros. You can choose which wins, and it's neat to debug.

Hear, hear!  This is exactly what I was trying to get at.  It will
*not* cause undue obfuscation and *will* enforce discipline.

BTW, Simon, I really want to collaborate with your Sapphire work or
merge it with the stuff I've been doing.  My files are not in any
compilable form at the moment, but you can see them using...

echo :pserver:[EMAIL PROTECTED]:/home/cvs A >> ~/.cvspass
mkdir pickle pickle/CVS
cd pickle
echo :pserver:[EMAIL PROTECTED]:/home/cvs >CVS/Root
echo pickle >CVS/Repository
touch CVS/Entries
cvs update

The basic methodology uses C++ inheritance from the Perl 5 structs
typedef'd in perl.h, such as struct interpreter (PerlInterpreter) and
struct sv (SV).  My subclasses add only methods, no members, and no
virtuals.  Thus, the results are binary-compatible with Perl 5.

Then I wrap the whole thing in some completely opaque structs to build
a library whose interface does not require any perl headers and lets
programs work with different configurations, such as MULTIPLICITY,
without recompilation.

-John

Re: inline mania

[John Tobey]

Sam Tregar <[EMAIL PROTECTED]> wrote:
> On Tue, 1 Aug 2000, John Tobey wrote:
> 
> > I propose that all internal and external functions be implemented in
> > header files and declared inline.  The external runtime API will be
> > generated by compiling non-inline wrappers.
> > 
> > This will make possible an interesting performance optimization with
> > GNU CC, whereby op_ppaddr is replaced by a local label, and runops is
> > a block of code that uses goto instead of function pointer calls.  It
> > could be kept clean (by Perl 5 standards) using techniques similar to
> > embed.pl's.
> 
> I counter-propose that we avoid early optimization.  I propose that our
> designs begin with readbility and modularity as their highest priorities.  
> Later we can optimize the sense out of it but at least it would be
> readable to start with.
> 
> I propose that we treat "clean (by Perl 5 standards)" as not clean enough
> for Perl 6.

Fair enough, but consider my proposal in two steps.  Concretely, we
should implement everything as inline from the start.  It is hardly
any more difficult to read functions declared inline than those that
are not.  The non-inline wrappers will have to be maintained, but they
will all be trivial and need only change when functions are added or
prototypes altered.  Heck, with a little discipline, they could be
generated like perlapi.c.

The first step (all-inline through wrappers) is not an optimization at
all, and therefore not an early optimization.  It allows for a lot of
future optimizations, though.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\

inline mania

[John Tobey]

I propose that all internal and external functions be implemented in
header files and declared inline.  The external runtime API will be
generated by compiling non-inline wrappers.

This will make possible an interesting performance optimization with
GNU CC, whereby op_ppaddr is replaced by a local label, and runops is
a block of code that uses goto instead of function pointer calls.  It
could be kept clean (by Perl 5 standards) using techniques similar to
embed.pl's.

-- 
John Tobey, late nite hacker <[EMAIL PROTECTED]>
\\\   ///
]]] With enough bugs, all eyes are shallow.   [[[
///   \\\