design of the Prelude (was Re: Rakudo leaving the Parrot nest)
Patrick R. Michaud wrote (on p6c): On Thu, Jan 15, 2009 at 08:53:33AM +0100, Moritz Lenz wrote: Another thing to keep in mind is that once we start to have a Perl 6 prelude, we might decide to be nice neighbors and share it with other implementations, as far as that's practical. My guess is that there will be a shared prelude that is maintained in a central repository like the spectests, but that individual implementations are likely to want or need customized versions of the prelude for performance or implementation reasons. In this sense the shared prelude acts as a "reference standard" that implementations can use directly or optimize as appropriate. Now about the Perl 6 Prelude, rather than have customized versions, ... A couple things I've learned from designing a language for implementation over multiple architectures are: 1. It is usually possible to define almost all of the built-in types and operators of a language in terms of each other using the mechanisms the language provides for user-defined types and operators; left to itself this means the language is recursively defined. 2. Each implementation architecture has its own concepts of what are the most fundamental and efficient types and operators to use as a foundation. For example, Haskell and Parrot would have many differences on what the fundamentals are. What I recommend, and forgive me if things already work this way, is to expand the Prelude so that it defines every Perl 6 core type and operator using pure Perl 6, complete enough such that there are as few as possible actual primitives not defined in terms of other things. This Prelude would then be shared unchanged between all the Perl 6 implementations. Then, each implementation would also define its own PreludeOverride file (name can be different) in which it lists a subset of the type and operator definitions in the Prelude that the particular implementation has its own implementation-specific version of, and the latter then takes precedence over the former in terms of being compiled and executed by the implementation. And so, as Perl 6 evolves or new implementations come along, the amount of work necessary specific to the implementation to just get Perl 6 running at all is minimal, and so more implementation specific work then is mainly for just making it run more efficiently. Take for example, non-integer numeric types and operators. These could be fully defined in the Prelude or other plain Perl 6 file in terms of the (big) Int type and its operators, so implementations only need native (big) Int support to get all the other numeric types working for free. Then, if some implementations want to get faster float or complex or small-int etc performance, they can override with versions that use C-defined or machine native etc types and operators in those cases. (As a more extreme example, you could also define character strings etc in terms of dense arrays of Perl Int, and all the complexity for Unicode etc would have a default implementation in Perl 6; in that case, Str, and Perl identifiers, wouldn't be opaque but rather would be class defs with array-of-Int attributes.) So abstractly speaking the Prelude is still customized, but all differences from the self-hosted shared one are in separate files. So have I made any sense here, and what do you think? -- Darren Duncan
Re: design of the Prelude (was Re: Rakudo leaving the Parrot nest)
On Thu, 2009-01-15 at 16:03 -0800, Darren Duncan wrote: > Patrick R. Michaud wrote (on p6c): > > On Thu, Jan 15, 2009 at 08:53:33AM +0100, Moritz Lenz wrote: > >> Another thing to keep in mind is that once we start to have a Perl 6 > >> prelude, we might decide to be nice neighbors and share it with other > >> implementations, as far as that's practical. > > > > My guess is that there will be a shared prelude that is maintained > > in a central repository like the spectests, but that individual > > implementations are likely to want or need customized versions of > > the prelude for performance or implementation reasons. In this > > sense the shared prelude acts as a "reference standard" that > > implementations can use directly or optimize as appropriate. > > What I recommend, and forgive me if things already work this way, is to > expand > the Prelude so that it defines every Perl 6 core type and operator using pure > Perl 6, complete enough such that there are as few as possible actual > primitives > not defined in terms of other things. This Prelude would then be shared > unchanged between all the Perl 6 implementations. > > Then, each implementation would also define its own PreludeOverride file > (name > can be different) in which it lists a subset of the type and operator > definitions in the Prelude that the particular implementation has its own > implementation-specific version of, and the latter then takes precedence over > the former in terms of being compiled and executed by the implementation. The problem with this method is that there are usually *several* ways to implement each feature in terms of some number of other features. The creators of the shared prelude are then stuck with the problem of deciding which of these to use. If their choices do not match the way a particular implementation is designed, it will then be necessary for the implementation to replace large swaths of the Prelude to get decent performance. For example, implementations in pure C, Common Lisp, and PIR will probably have VASTLY different concepts of available and optimized primitive operations. A prelude written with any one of them in mind may well be pessimal for one of the others. That's not to say it's not a useful idea for helping to jumpstart new implementations -- I just somewhat doubt that a mature implementation will be able to use more than a fraction of a "common" prelude. -'f P.S. I did this sort of thing once -- a Forth prelude that attempted to minimize the primitive set, and it *was* very nice from an abstract perspective. Unfortunately, it also made some operations take millions of cycles that would take no more than one assembly instruction on just about every CPU known to man. It's a REALLY easy trap to fall into.
Re: design of the Prelude (was Re: Rakudo leaving the Parrot nest)
Geoffrey Broadwell wrote: The problem with this method is that there are usually *several* ways to implement each feature in terms of some number of other features. The creators of the shared prelude are then stuck with the problem of deciding which of these to use. If their choices do not match the way a particular implementation is designed, it will then be necessary for the implementation to replace large swaths of the Prelude to get decent performance. For example, implementations in pure C, Common Lisp, and PIR will probably have VASTLY different concepts of available and optimized primitive operations. A prelude written with any one of them in mind may well be pessimal for one of the others. That's not to say it's not a useful idea for helping to jumpstart new implementations -- I just somewhat doubt that a mature implementation will be able to use more than a fraction of a "common" prelude. I have a few answers to this: 0. I agree that, no matter what, the implementation will still want to substitute in its own versions. But so what? Having a reasonably more complete high-level reference implementation of the Prelude in Perl 6 won't lose us anything over what we have now and should on average gain something. 1. What we *should* be doing with the Prelude, like with STD.pm, is write under the assumption that the implementation is also written in Perl 6. We should write the Prelude in as declarative a manner as possible, saying *what* we want to happen rather than how, such as you do when writing in a functional language. We should make use of Perl's higher-level tools like hyper-operators and reduce-operators etc and write in a fashion that is developer focused, same as writing normal Perl 6. We do not, except where it makes sense, want to be defining things in terms of the lowest level things possible, as that would be premature optimization, which may help some implementations and harm others. We should instead be defining all the low level operators in terms of the high level operators, where possible. It is easier for an average implementation to translate a high level source operation to low level native operations on average, than try to amalgamate a whole bunch of low level source operations to fewer high level implementation operations. (Note for example I suggested using big/unlimited-size Int as a basis of definition rather than a machine-int.) Don't be afraid to be recursive, if it is even possible. For example, one could define Hash in terms of Array *and* define Array in terms of Hash. Or Int in terms of Rat *and* Rat in terms of Int. 2. We should be able to live with the benefit of at least short term hindsight, seeing what likely implementations we will have, and aim for the middle. For example, write as if high-level concepts are supported in the implementation. 3. Perl 6 does have multi-methods. Maybe make use of them to define alternative sample implementations where it makes sense, though don't go too far citing combinational exponents. Or if multi-methods don't work quite that way, we could add a kind of trait to them or something that says use one or the other but not both, and implementations can mark in their override file to pick a particular version. Even if not all implementations are the same, some are similar to each other and can share that work. Call them "possible representations" or "possible implementation versions" or something. To some extent I think Perl 6 does have what we need in a different fashion, such as where you can declare a class or attribute and indicate an implementation type like Opaque vs Hash or what-have-you. P.S. I did this sort of thing once -- a Forth prelude that attempted to minimize the primitive set, and it *was* very nice from an abstract perspective. Unfortunately, it also made some operations take millions of cycles that would take no more than one assembly instruction on just about every CPU known to man. It's a REALLY easy trap to fall into. That may be because you wrote in terms of a few low level operations rather than a few high level ones; what if you did the reverse? As for me, I am in the process of doing this now with my Muldis D language, which is fairly high level and should run on everything Perl 6 does, though with a stronger implied support for functional-paradigm-supporting languages, and also run over SQL; though at the same time each implementation can choose to leave out some features as it sees fit, some are more important to include than others. In my implementation of Muldis D over Perl (Muldis Rosetta's default engine), most built-in types and operators are defined as users would, in terms of other ones, save for its only 4 truly-primitive types, [Int (opaque unlim-size scalar), String (dense-array-of-Int), QTuple (heterogenous collection), QRelation (homogenous collection)], but the Perl class/object representing a user/as-
Re: design of the Prelude (was Re: Rakudo leaving the Parrot nest)
Forgive my ignorance, but what is a Prelude? -- Jonathan "Dataweaver" Lang
Re: design of the Prelude (was Re: Rakudo leaving the Parrot nest)
Jon Lang wrote: Forgive my ignorance, but what is a Prelude? The Prelude is a file written in Perl 6 that defines some Perl 6 built-ins. See http://perlcabal.org/svn/pugs/view/src/perl6/Prelude.pm for what AFAIK is the newest version. -- Darren Duncan
Re: design of the Prelude (was Re: Rakudo leaving the Parrot nest)
On Thu, Jan 15, 2009 at 6:45 PM, Jonathan Scott Duff wrote: > On Thu, Jan 15, 2009 at 8:31 PM, Jon Lang wrote: >> >> Forgive my ignorance, but what is a Prelude? >> >> -- >> Jonathan "Dataweaver" Lang > > The stuff you load (and execute) to bootstrap the language into utility on > each invocation. Usually it's written in terms of the language you're > trying to bootstrap as much as possible with just a few primitives to get > things started. OK, then. If I'm understanding this correctly, the problem being raised has to do with deciding which language features to treat as primitives and which ones to bootstrap from those primitives. The difficulty is that different compilers provide different sets of primitives; and you're looking for a way to avoid having to write a whole new Prelude for each compiler. Correct? Note my use of the term "language features" in the above. Presumably, you're going to have to decide on some primitive functions as well as some primitive datatypes, etc. For instance: before you can use meta-operators in the Prelude, you're going to have to define them in terms of some choice of primitive functions - and that choice is likely to be compiler-specific. So how is that any easier to address than the matter of defining datatypes? Or is it? -- Jonathan "Dataweaver" Lang
Re: design of the Prelude (was Re: Rakudo leaving the Parrot nest)
On Thu, Jan 15, 2009 at 8:59 PM, Jon Lang wrote: > OK, then. If I'm understanding this correctly, the problem being > raised has to do with deciding which language features to treat as > primitives and which ones to bootstrap from those primitives. The > difficulty is that different compilers provide different sets of > primitives; and you're looking for a way to avoid having to write a > whole new Prelude for each compiler. Correct? > > Note my use of the term "language features" in the above. Presumably, > you're going to have to decide on some primitive functions as well as > some primitive datatypes, etc. For instance: before you can use > meta-operators in the Prelude, you're going to have to define them in > terms of some choice of primitive functions - and that choice is > likely to be compiler-specific. So how is that any easier to address > than the matter of defining datatypes? Or is it? Did I miss something here? I've never heard Prelude. I'm not really convinced that each implementation should have a large set of shared code; that seems contrary of the idea of having independent implementations. Having to support a common set of implemented classes like this may end up being more of a burden than a benefit. You may find each implementation replacing parts of the Prelude to serve their needs. It also seems like that Prelude.pm is dated and also very pugs specific, which is ironic. What are all the references to Pugs::Internals and pugs_internals_m:perl5? Is rx:Perl5 and rd:P5 valid perl6? I'm skeptical of the this idea unless someone can convince me otherwise. -Jason "s1n" Switzer
Re: design of the Prelude (was Re: Rakudo leaving the Parrot nest)
Following some responses I've seen, I'll try to clarify my proposal. Basically its like this. A significant subset of Perl 6 native features, eg types and operators, native meaning they are declared and described in the Perl 6 Synopsis documents, have been implemented under Pugs by being written in plain Perl 6 in terms of other Perl 6 features, rather than being written in Haskell or C or Perl 5. See for example the modules Math::Basic and Set, which live in the ext/ directory of Pugs and are defined like any user-defined module. Then recall that different languages have varying concepts of what constitutes a core language feature and what is an optional or third-party extension feature. For example, some languages have native types and operators for temporal artifacts like dates and times and durations, and others don't. Some languages have "big" numbers as a fundamental and others only have machine-size numbers. Now I anticipate that for any given native / Synopsis-described types and operators of Perl 6, some Perl 6 implementations will define those as plain Perl 6 code in terms of other Perl 6 features, and other implementations will instead have them directly wrap features of the host language. My main point here is that the demarcation line between what is written in Perl 6 and what is written in the host language can vary wildly and can pay little to no attention to the conceptual boundaries in the Perl 6 Synopsis between what is more fundamental and what is more of an extension. Therefore, it does not hurt to increase as much as possible the fraction of the Perl 6 language that has a reference implementation defined just in terms of Perl 6. Especially if this reference Perl 6 code is written in a more declarative fashion, it increases the likelihood that more starter code is available to help bootstrap any given Perl 6 implementation, where writing some parts of the language are more optional for them to do themselves than otherwise. Note that I just referred to this body as the Prelude because I was replying to a particular p6c comment naming that, and also Pugs had something by that name with a similar purpose which is what was referred to in the original post. So I'm not so much talking about the existing Prelude file as the concept it represents, which is making it easier to share code between Perl 6 implementations, where each implementation wants to use it. Or just to take advantage of the fact that Perl 6 itself should be easier to write some kinds of code in than other languages, including itself. We can go further than the minimal bit we have now. -- Darren Duncan
Operator sleuthing...
I'm re-working my "Periodic Table of the Operators" chart to be up-to- date. I did the first major pass based on S03-operators. However, the last few days I've been plowing through STD.pm and have discovered that there some differences. Since STD.pm is considered more up to date, I'll be changing the chart to match that. However, I thought it would be useful to share the detailed list of differences I found. Besides, there are some embedded questions in there, I'd love to have answered. Thanks, - Mark STD.pm: r24855 | lwall | 2009-01-10 S03-operators.pod r24895 | lwall | 2009-01-13 == Associativity == Prefix and postfix precedence levels have no associativity in STD, but are left (mostly) or right in S03. Levels affected are: %methodcall %autoincrement %symbolic_unary %named_unary %loose_unary %list_prefix Some levels have left associativity in STD, but are list in S03. Levels affected: %concatenation %tight_and %tight_or %loose_and %loose_or Oddly, sym<^^> ( --> Tight_or) is forced to :assoc in STD. ?? Even though for most of these operators there is no computational difference between left and list assoc, list seems more "in the spirit" to me. I plan on leaving them listed as 'list' assoc in the chart unless there is a reason to make them left. Conditional level is right in STD, and left in S03. == Added == STD has a new Methodcall operator construction: token privop ( --> Methodcall) { '!' } ?? What is this? STD has postfix:sym. I understand this is the imaginary operator. STD has infix:sym<.=>. I understand this to be the method call assigning form when there is space after the operator. ?? Is this right? STD has <==, ==>, <<==, and ==>> as true operators. S03 groups them as terminators. STD has sym<;> as both an infix operator ( --> Sequencer), and as a terminator. ?? Which is it? Since I think most people think of it as a statement terminator, I plan on leaving it off the chart. == Removed == STD is missing sym<\\> at Symbolic_unary level. ?? Did capture cease to be an operator? STD is missing sym at Multiplicative level. ?? I'm assuming this is just an oversight. == Misc == The ff and fff family are at %nonchaining precedence in STD, but at %conditional precedence in S03. STD has infix:sym and infix:sym each declared as tokens twice. I'm assuming this is just some accidental duplication. At Loose_unary, STD has which seems more general than the operator adverbs discussed in S03 at Loose unary precedence. The infix:sym<:> invocant marker is at Comma precedence in STD, but List prefix in S03. It is also left assoc. in STD, but right in S03. The sigils used as operators at List_prefix in STD include sym<&>. It isn't listed in S03. ?? Should it be included as an operator, or is it senseless as an operator? S03 lists the a number of named listops: all one any none die warn fail item list slice hash These aren't coded expressly in STD, presumably as they are all handled by the generic named listop code. ?? Is there any reason to call them out in a list of operators as S03 does? Or is it just that these named listops happen to be the same as existing symbolic ones and are worth pointing out. The named listops seem to be handled in STD by token term:identifier ( --> Term ) But this is at Term precendence, not List_prefix. ?? Did these move? The ! metaop in STD requires that the op be either chaining or have an associativity (any) and be :bool. But %chaining is the only set of operators that has :bool... ?? Is this just "belt and suspenders" checking, or can that meta op apply to more? Mark Lentczner http://www.ozonehouse.com/mark/ m...@glyphic.com
Re: Operator sleuthing...
Mark Lentczner wrote: > STD has sym<;> as both an infix operator ( --> Sequencer), and as a > terminator. > ?? Which is it? Since I think most people think of it as a statement > terminator, I plan on leaving it off the chart. It is both. Examples where it is used as an infix operator include: loop (my $i = 1; $i < 10; $i *= 2) { ... } my @@slice = (1, 2; 3, 4; 5, 6) Presumably, Perl is capable of distinguishing the meanings based on what the parser is expecting when it finds the semicolon. -- Jonathan "Dataweaver" Lang
r24926 - docs/Perl6/Spec src/perl6 t/operators t/run
Author: lwall Date: 2009-01-16 08:42:00 +0100 (Fri, 16 Jan 2009) New Revision: 24926 Modified: docs/Perl6/Spec/S03-operators.pod src/perl6/STD.pm t/operators/adverbial_modifiers.t t/run/02-dash-n.t t/run/03-dash-p.t Log: [STD] alignment with S03 on associativity noted by mtnviewmark++ [STD] added various missing functions and types [t/spec] remove my() function calls :) Modified: docs/Perl6/Spec/S03-operators.pod === --- docs/Perl6/Spec/S03-operators.pod 2009-01-16 06:07:31 UTC (rev 24925) +++ docs/Perl6/Spec/S03-operators.pod 2009-01-16 07:42:00 UTC (rev 24926) @@ -45,7 +45,7 @@ C Chaining infix!= == < <= > >= eq ne lt le gt ge ~~ === eqv !eqv X Tight and && X Tight or || ^^ // min max -L Conditional ?? !! ff fff +R Conditional ?? !! ff fff R Item assignment = := ::= => += -= **= xx= .= L Loose unary true not :by(2) X Comma operator, p5=> @@ -53,7 +53,8 @@ R List prefix : print push say die map substr ... [+] [*] any $ @ X Loose and and andthen X Loose or or xor orelse -N Terminator; <==, ==>, <<==, ==>>, {...}, unless, extra ), ], } +L Sequencer <==, ==>, <<==, ==>> +N Terminator; {...}, unless, extra ), ], } Using two C symbols below generically to represent any pair of operators that have the same precedence, the associativities specified above Modified: src/perl6/STD.pm === --- src/perl6/STD.pm2009-01-16 06:07:31 UTC (rev 24925) +++ src/perl6/STD.pm2009-01-16 07:42:00 UTC (rev 24926) @@ -97,7 +97,7 @@ bit bool Order Increasing Decreasing -Ordered Callable Positional Associatve +Ordered Callable Positional Associative Abstraction Ordering KeyExtractor Comparator OrderingPair IO @@ -124,8 +124,9 @@ # XXX likewise for routine defs my @baseroutinenames = qw[ -WHAT WHICH VAR +WHAT WHERE HOW WHICH VAR WHO WHENCE new any all none one +not true die exit warn caller want @@ -136,7 +137,7 @@ cat classify quotemeta chr ord -p5chop chop p5chomp chomp +p5chop chop p5chomp chomp trim index rindex substr join split comb pack unpack uc ucfirst lc lcfirst @@ -148,6 +149,7 @@ say print open close printf sprintf slurp unlink link symlink elems grep map first reduce sort uniq push reverse take splice +lines getc zip each roundrobin caller return leave pop shift unshift reduce @@ -224,14 +226,14 @@ constant %multiplicative = (:prec, :assoc, :assign); constant %additive= (:prec, :assoc, :assign); constant %replication = (:prec, :assoc, :assign); -constant %concatenation = (:prec, :assoc, :assign); +constant %concatenation = (:prec, :assoc, :assign); constant %junctive_and= (:prec, :assoc, :assign); constant %junctive_or = (:prec, :assoc, :assign); constant %named_unary = (:prec); constant %nonchaining = (:prec, :assoc); constant %chaining= (:prec, :assoc, :bool); -constant %tight_and = (:prec, :assoc, :assign); -constant %tight_or= (:prec, :assoc, :assign); +constant %tight_and = (:prec, :assoc, :assign); +constant %tight_or= (:prec, :assoc, :assign); constant %conditional = (:prec, :assoc); constant %item_assignment = (:prec, :assoc); constant %loose_unary = (:prec); @@ -239,8 +241,8 @@ constant %list_infix = (:prec, :assoc, :assign); constant %list_assignment = (:prec, :sub, :assoc); constant %list_prefix = (:prec); -constant %loose_and = (:prec, :assoc, :assign); -constant %loose_or= (:prec, :assoc, :assign); +constant %loose_and = (:prec, :assoc, :assign); +constant %loose_or= (:prec, :assoc, :assign); constant %sequencer = (:prec, :assoc, :nextterm); constant %LOOSEST = (:prec); constant %terminator = (:prec, :assoc); Modified: t/operators/adverbial_modifiers.t === --- t/operators/adverbial_modifiers.t 2009-01-16 06:07:31 UTC (rev 24925) +++ t/operators/adverbial_modifiers.t 2009-01-16 07:42:00 UTC (rev 24926) @@ -31,7 +31,7 @@ my $bar = 123; my @many = (4,5); sub dostuff(){"stuff"} - my($v,$e); + my ($v,$e); $e = (foo => $bar); $v = :foo($bar); is ~$v, ~$e, ':foo($bar)'; Modified: t/run/02-dash-n.t === --- t/run/02-dash-n.t 2009-01-16 06:07:31 UTC (rev 24925) +++ t/run/02-dash-n.t 2009-01-16 07:42:00 UTC (rev 24926) @@ -37,7 +37,7 @@ "; sub nonce () { return (".{$*PID}." ~ (1..1000).pick) } -my($in_fn, $out_fn) = >>~<< nonce; +my ($in_fn, $out_fn) = >>~<< nonce; my $h = open("$in_fn", :w); $h.print($str); $h.close(); Modified: t/run/