Re: Some namespace notes
Thus wrate Dan Sugalski: At 10:13 AM -0800 1/13/04, Jeff Clites wrote: Short version: I was originally going to argue for fully hierarchical namespaces, identified as above, but after turning this over in my head for a while, I came to the conclusion that namespaces are not conceptually hierarchical (especially as used in languages such as Perl5 and Java, at least), so I'm going to argue for a single string (rather than an array) as a namespace identifier. It is probably worth pointing out that even in Perl5 namespaces are implemented as though they were hierarchical. They're passed around as strings ("Foo::Bar::baz"), but when a variable is fetched (C in gv.c), perl parses the string as though it was a multidimensional key, with dimensions separated by "::" "'". As Uri pointed out, this makes it easy to take and pass around references to stashes; it also reduces memory usage, since there is only one copy each of "Foo" and "Bar", regardless of how many items are in %Foo::Bar:: . Performance-wise, I would guesstimate that it's more-or-less a wash between parsing strings and parsing multidimensional keys, so as long as we precreate the keys (keep thm in a constant table or something), I see no performance issues. Just keep the variable's name separate from its namespace (as codified in pdd06), and IMHO we should be golden with multidimensional keys. Here's my big, and in fact *only*, reason to go hierarchical: We don't need to mess around with separator character substitution. Other than that I don't much care and, as you've pointed out, most of the languages don't really do a hierarchical structure as such. Going hierarchical, though, means we don't have to do "::"/":"/"/"/whatever substitutions to present a unified view of the global namespaces. But we do need to worry about cross-language namespace collisions... :-D -- BKS
Re: Vtables organization
Dan Sugalski wrote: I was going to go on about a few ways to do this, but after I did I realized that only one option is viable. So, let's try this on for size: Vtables are chained. That means each vtable has a link to the next in the chain. It *also* means that each call into a vtable function has to pass in a pointer to the vtable the call came from so calls can be delegated properly. If we don't want this to suck down huge amounts of memory it also means that the vtable needs to be split into a vtable header and vtable function table body. Downside there is that we have an extra parameter (somewhat pricey) to all the vtable functions. This is sort of icky. What about dynamically constructing vtables and caching them? We'd probably need some sort of mangling scheme, but it wouldn't be too hard; maybe just the ordered list of packages separated by NULs or something. Then again, though, any sort of stacking runs into data collisions in the PMC. So maybe we just need lots of vtable types. Ties should probably be their own vtable (or one each for tied scalars, hashes, and arrays); read-only-ification should maybe be a special case that actually does dynamically create (and cache, if need be) vtables, replacing ->set_* and ->morph with functions that throw an exception. For thread-sharing: proxies seem to be standard? What other attributes or traits or whatever actually expect to still be able to directly access the underlying variable? (Ties and overloads, for instance, _don't_ - they have to provide their own backing storage.) I suppose what I'm getting down to is that the intelligence should just be in the compile-time PMC class generator, not dynamic in the runtime. Except maybe for read-only-ification. -- BKS
Re: Vtables organization
Thusly did Dan Sugalski inscribe: At 3:33 PM -0500 1/16/04, Benjamin K. Stuhl wrote: Dan Sugalski wrote: I was going to go on about a few ways to do this, but after I did I realized that only one option is viable. So, let's try this on for size: Vtables are chained. That means each vtable has a link to the next in the chain. It *also* means that each call into a vtable function has to pass in a pointer to the vtable the call came from so calls can be delegated properly. If we don't want this to suck down huge amounts of memory it also means that the vtable needs to be split into a vtable header and vtable function table body. Downside there is that we have an extra parameter (somewhat pricey) to all the vtable functions. This is sort of icky. What about dynamically constructing vtables and caching them? How would one wrap a vtable, then? If you do this, there's no way to call back into the original (or earlier wrapping) entry. Other than the special case of :readonly, can you give me an example of when you'd need to, rather than simply writing a PMC class that inherits from some base? I'm having trouble thinking of an example of when you'd want to be able to do that... After all, since operator overloading and tying effectively _replace_ the builtin operations, what more does one need? -- BKS
Re: Vtables organization
Luke Palmer wrote: Benjamin K. Stuhl writes: Other than the special case of :readonly, can you give me an example of when you'd need to, rather than simply writing a PMC class that inherits from some base? I'm having trouble thinking of an example of when you'd want to be able to do that... After all, since operator overloading and tying effectively _replace_ the builtin operations, what more does one need? Well, other than Constant, we need to be able to put locking on shared PMCs. We'd like to add a debug trace to a PMC. We could even make any kind of PMC sync itself with an external source on access, though that's a bit of a pathological case. Indeed, all of this can be done, however, by subclassing Ref. I think the reason this isn't sufficient is that we want to change the actual PMC into this new variant when it is possibly already in existence. Like my C was evilly trying to do. Perhaps there's a way to get that working safely... The issue is that the PMC's original vtable assumes (and should, IMHO be _able_ to assume) that it has total control over the PMC's data, so there is nowhere in the PMC to put a lock or a handle to an external source or anything. So you'd either need a Ref of some sort anyway or a global lookup table, which seems to be an even worse idea. Debug tracing, though, is a good question... I hate to pass an extra pointer to every vtable call just for that, though... -- BKS
Re: Vtables organization
Dan Sugalski wrote: [sniping to reduce verbiage] The issue is that the PMC's original vtable assumes (and should, IMHO be _able_ to assume) that it has total control over the PMC's data, Well... I think I'll disagree here. The *class* vtable can assume this. However that doesn't mean that any random vtable function can. Yes, that's what I meant by the original vtable. :-) In addition to the thread autolocking front end and debugging front end vtable functions, both of which can be generic, there's the potential for tracing and auditing front end functions, input data massaging wrappers, and all manner of Truly Evil front (and back) end wrappers that don't need to actually access the guts of the PMC, but can instead rely on the other vtable functions to get the information that they need to operate. Not that this necessarily mandates passing in the vtable pointer to the functions, but the uses aren't exactly marginal. Going back to the idea of generating these vtables on the fly (and caching them): each instance of a vtable gets a void* closure in the vtable itself, so at a certain expense in extra vtables, one could hang a structure off of that that includes a pointer to the original vtable. E.g. (pseudo-code) if (we don't have a tracing PerlInt in our cache) { TracePerlIntVtable = clone_vtable(interp, TraceVtable); vtable_set_data(interp, TracePerlIntVtable, PerlIntVtable); cache(TracePerlIntVtable); } TraceVtable::get_number(INTERP, self) { FLOATVAL f; // I don't have the headers in front of me to get the right field names... VTABLE *my_vtbl = self->vtable; VTABLE *old_vtbl = my_vtbl->private; self->vtable = old_vtbl; f = self->vtable->get_number(interp, self); TRACE("%p->get_number() = %f", self, f); self->vtable = my_vtbl; return f; } With slightly more complicated closures, most of the listed uses can use this method. It's memory-heavy, but not too bad if we need a fair number of PMCs of each wrapped type and the cost gets amortized over them. And this does save an argument to every PMC function and just makes the wrapping classes pay. Where it's suboptimal is for classes like autolocking that really want a closure of their own on every PMC. But then again, that would again probably get into issues of data collisions (except for locking, which has its own reserved area). Just some thoughts. -- BKS
Re: Layering PMCs
Dan Sugalski wrote: Okay, time to think about this. We need the ability to layer PMCs. Nothing new, we need something of the sort for transparent read-only-ness and probably thread-safety (though we don't have to do it that way) and folks are going to want to do undoable custom vtable layering. While we don't *have* to let 'em do that, I'm inclined to if only because a good general-purpose mechanism will make things less error-prone overall. So, then, the question is... how? I've got two schemes, neither of which I completely like, so I'm up for some discussion and more ideas. The second scheme's major advantage (preserving the address of the logical (user-code-visible) PMC, so we don't have to run around updating symbol tables, or worse yet arrays, every time we wrap a PMC) seems IMHO to be pretty overwhelming. AFAICT, it shouldn't really matter much that peeks at the logical PMC's internals may be incorrect (with one major caveat: if the _vtable_ functions try to use the PMC's address as an index into some data structure, things will go wrong since the PMC the base vtable functions get no longer has the same address as the logical PMC; if external things want to do so, that's fine though, since the logical PMC's address hasn't changed). After all, AIUI no one but the vtable functions is supposed to be poking inside the PMC, and the wrapper PMC (which happens to have the same address as the original PMC) will of course pass the new wrapped PMC down when it redispatches. (by MMD, I trust? Note that I think this scheme requires that the wrapper PMC types dispatch like the _base_ type (the type at the bottom of the layer stack, the original PMC) that they are wrapping (which means dynamic vtable and multimethod generation for each new wrapper/base type combo or some such so that the types work out correctl).) This should even work correctly for things like Continuations as long as the opcodes don't try to be too smart. And I can definitely see (auto-)wrapping Subs as possibly being a reasonable (not too much code-duplication) way to implement privilige-domain-crossing checks for a secure Parrot. -- BKS
Re: Functions for embedders to override
Dan Sugalski wrote: Since we're running into Ponie issues with this, which means we'll run into Apache issues as well as any number of other systems When Parrot's being embedded I can see the following functions needing overriding by the embedder: *) Memory: malloc, realloc, calloc, free *) Signals: handler register, Handler un-register, signal raise, alarm set *) Files: Open, close, seek, tell, read, write. (Possibly asynchronously) *) Environment: getenv, setenv *) Threading/Process manipulation? (fork, spawn, wait, pthread_*) (Consider an application built on a runtime that expects to know about all threads in a process, or an embedder who wants to let their Parrot programs call back into the main program or integrate into their main event loop.) * Networking: socket, accept, connect, listen, etc. (see "Files") A good place to look at for the complete list is Perl 5's system abstraction layer. -- BKS