I am pretty worried about the @suspendable annotation. The way this shift/reset
stuff works is it modifies the scala compiler to do something called
continuation passing style. aka CPS.
I'd be ok if that was isolated to just a segment of the code. Maybe there is
some natural way to do that?
But it seems to me that all code on the pathway from where a reset block is
entered to where a shift is called, all of it has to propagate this
@suspendable behavior and be compiled by way of this CPS plug in. That looks ok
for the tiny toy examples, but for a giant code base like Daffodil runtime1
unparser, that seems fragile, potentially has impact on debugging, memory
allocation, and performance of the code, and,... well given the lack of
enthusiastic support for shift/reset I think it is risky.
The only other option I can think of is to spawn a separate thread, allow true
concurrency in a producer-consumer model.
We already have a Coroutines library you may recall. We're not using it in the
code base now, and it's fairly high-overhead as it is a depth 1 queue, so is
constantly switching threads. It might have better performance characteristics
if the switching was reduced to once every 100 events or similar. Streaming
behavior does not have to switch from events to pull at granularity 1 event per
pull, it can be much coarser than that to push overhead down.
The limiting thing here really seems to be the JVM. Java virtual machines
simply don't support the concept of co-routines in any sensible manner.
There are also some coroutine-style libraries for Java that depend on byte-code
modification. I suspect those have a similar issue to the CPS transformation,
ie., all the code on the way to a suspension requires the byte code
modification, but I may be wrong.
From: Steve Lawrence
Sent: Thursday, April 23, 2020 11:21 AM
To: dev@daffodil.apache.org
Subject: Re: Coroutines - was Re: Daffodil SAX API Proposal
Thanks Mike! Continuations seems like a better alternative, at least
from a support point of view. Though, it's a little concerning that no
one is really stepping up to port it to 2.13, but I don't think we're in
any rush to get to 2.13. And I personally find the reset/shift concept a
bit harder to wrap my head around than the co-routine resume/yield, but
ultimately it's not too bad.
To see how it would work with our DataProcessor/InfosetInputter, I
forked and updated your gist to include things like InfosetInputters,
DataProcessor, ContentHandler, etc. and added a bunch of println's and
comments to make sure things were behaving the way I thought they should.
https://gist.github.com/stevedlawrence/5e16081f4690448de6131af02daacea9
I think it came out pretty straightforward. I also modified this so that
there isn't as much back and forth between hasNext/next like I have in
the current proposal. The only time we go back the to
ContentHandler/producer is when next() is called, and we only go back to
the InfosetInputter/consumer when a complete event is found, including
hasNext.
I do have one concern with this approach. Scala required the
@suspendable annotation on the unparse() method of the DataProcessor and
on the next() method of the InfosetInputter for both the abstract class
and concrete SAX implementation. I'm not sure if that annotation causes
any problems when not used inside a reset block (i.e. old API style), or
if that annotation will end up cascading throughout the codebase. Seems
like there's a possibility for that to happen. Maybe I just need to
reorganize the code a bit, but it's not clear to me how.
On 4/22/20 7:18 PM, Beckerle, Mike wrote:
> scala continuations is supported on 2.11 and 2.12, but work in progress for
> 2.13. The main web page for it says it is looking for a lead developer and
> without that typesafe/lightbeam is doing bare minimum maintenance.
>
> A producer/consumer idiom like what we need is easily expressed using this
> shift/reset thing.
>
> Here's a gist that does a control turnaround from a handler to a
> pull-oriented while loop. Took me a bit of research to get the build.sbt
> right so this would "just work"
>
> https://gist.github.com/mbeckerle/4c1d8f8c365958ef7d01bf770fa6317c
>
>
>
> From: Beckerle, Mike
> Sent: Wednesday, April 22, 2020 5:01 PM
> To: dev@daffodil.apache.org
> Subject: Re: Daffodil SAX API Proposal
>
> Another possibility is scala-asynch which I think can do what we want.
>
> From: Beckerle, Mike
> Sent: Wednesday, April 22, 2020 4:34 PM
> To: dev@daffodil.apache.org
> Subject: Re: Daffodil SAX API Proposal
>
> The alternative is probably scala.util.continuations aka "shift and reset".
>
> It's much harder to understand and use, but at least its in the standard
> library so is supported. (I think.)
>
>
> From: Steve Lawrence
> Sent: Wednesday, April 22, 2020 3:40 PM
> To: