Right. Macro definition ends, you subclass off the parser object, then immediately call into it
...
You, as a user-level programmer, don't have to track the state. The parser code will, but that's not a big deal.
OK, I suppose that works although that still means you're moving the complexity from the perl implementation to its usage: in this case, the perl 6 parser which is written in perl 6 -- but I can well imagine other people want to do the same, and they'll have to do a similar hack.
I really don't like that, perl normally moves the complexity away from the programmer and into perl.
They should be though, if a variable was hypothesized when the continuation was taken, then it should be hypothesized when that continuation is invoked.
Should they? Does hypotheticalization count as data modification (in which case it shouldn't) or control modification (in which case it should),
Isn't that the whole point of hypotheses in perl 6? You talk about "successful" and "unsuccessful" de-hypothesizing, and about abormals exits etc.. you seem to have a much complexer model of hypotheses than what's in my head.
I could be entirely missing things ofcourse, but I haven't seen any evidence to that yet.
To me, what 'let' does is temporize a variable except it doesn't get restored if you leave the scope, but only if you use a continuation to go back to a point where it wasn't hypothesized yet.
When the last continuation taken *before* the hypothesis is gone, so is the old version and thus the hypothesized variable becomes permanent.
The behavior regarding coroutines followed naturally from this, and so does the behavior inside regexen if they use the "continuation" semantics for backtracking -- which is what I'm suggesting.
This leave only behavior regarding preemptive threads, which is actually very easy to solve: disallow hypothesizing shared variables -- it simply makes no sense to do that. Now that I think of it, temporizing shared variables is equally bad news, so this isn't something new.
(Which makes continuations potentially more expensive as you need to then save off more info so on invocation you can restore the hypothetical state)
Actually, I think 'let' can handle this.. it's only invocation of continuations that will become more expensive because it needs to deal with the hypothesized variables
What about co-routines, then? And does a yield from a coroutine count as normal or abnormal exit for pushing of hypothetical state outward, or doesn't it count at all?
Your terminology gets rather foreign to me at this point. Assuming a co-routine is implemented using continuations, their behavior follows directly from the description above, and I think the resulting behavior looks fine. I don't see why people would hypothesize variables inside a co-routine anyway.
I hypotheticalize the variables. I then take a continuation. Flow continues normally, exits off the end normally, hypothetical values get pushed out. I invoke the continuation, flow continues, exits normally. Do I push the values out again?
If it ends normally, the variable isn't de-hypothesized at all. Also, the continuation was created *after* you hypothesized the variable, so when you invoke it nothing will happen to the variable.
How? Successfully or unsuccessfully? Does it even *count* as an exit at all if there's a pending continuation that could potentially exit the hypotheticalizing block later?
You're making 0% sense to me, apparently because your mental model of hypothesizing differs radically from mine.
Why? That doesn't make much sense, really.
Probably the same problem in opposite direction :-)
(And scare up a dual or better processor machine and I'll blow the doors off a cooperative threading scheme, synchronization overhead or not)
Ofcourse, for CPU-intensive applications that spread their computation over multiple threads on a multi-processor machine, you'll certainly need preemptive multithreading.
When exactly is the last time you wrote such an application in perl? :-)
Seriously though, I think in the common case cooperative threading is likely to be superior.. it has low overhead, it should have faster context switch time, you have no synchronization issues, and you can mostly avoid the need for explicit yielding: in many applications threads will regularly block on something anyway (which will yield to another thread)
But anyway, this is getting off-topic.. I'll save it for later. Regex first
No. Honestly I still don't see the *point*, certainly not in regards to regular expressions and rules. The hypothetical issues need dealing with in general for threads, coroutines, and continuations, but I don't see how any of this brings anything to rules for the parsing engine.
The flow control semantics the regex/parser needs to deal with are small and simple. I just don't see the point of trying to make it more complex.
More complex ?!
What I'm suggesting is a simple definition of hypothetical variables which makes backtracking easy to do using continuation in the general case, and therefore automatically also by rules.
Rules can then probably be optimized to *avoid* explicitly use continuation to the point where they have the speed you demand, while still keeping up appearances of the simple continuation-based backtracking semantics.
We're not backtracking with continuations, though.
I'm suggesting you do "officially", but optimize it away behind the scenes. This leaves nice and simple semantics for backtracking in general, while in fact their implementation inside rules is simple and efficient.
I have to admit I'm not 100% sure this is possible, but give me some time to try to work out the details :-)
-- Matthijs van Duin -- May the Forth be with you!
