So are you suggesting that the suspension that determines if a separator should be laid down *also* determines and lays down necessary alignment? So the grammar changes so that separators no longer include MTA during unparse, and that instead is handled by the various separator unparsers and the suspensions they create?
If if we come across other issues of nested suspensions the general solution is likely to combine the separate suspensions into a single suspension? On 9/24/20 2:12 PM, Beckerle, Mike wrote: > > I think there is an invariant that inserting these alignment suspensions is > violating. > > The chain of connected DOSs, well... information propagates across them. Any > time a later one in the chain wants to know information derived from prior > ones, a request ripples back the chain. So there is an assumption that you > can't insert a split, except at the end of the chain. > > That kind of lazy propagation algorithm is incompatible with there being > timing issues associated with inserting new splits after such information has > been requested. > > So I think what we need to do is insert, unconditionally, a split for a DOS > associated with the conditional separator, where that DOS will be there in > the chain, but will either behave as 0 length if the separator proves to be > suppressed, or behaves like an MTA if the separator proves to be > not-suppressed. > > Does that make sense? > > Note that in most text-centric formats, optimizers are supposed to be > optimizing out these MTA, since a format that is all text doesn't need them, > so this insertion of MTA, or these just suggested > PossiblySuppressedConditionalMTAs, should not be needed except in mixed > text+binary or multiple text encoding situations, which are rare. > > I do suspect these optimizations aren't working right currently though, so > we're seeing these MTA and other alignment regions being inserted more > frequently than they should. (Which has performance implications also....) > > > > > > > ________________________________ > From: Steve Lawrence <slawre...@apache.org> > Sent: Thursday, September 24, 2020 1:58 PM > To: dev@daffodil.apache.org <dev@daffodil.apache.org> > Subject: Handling Nested Suspensions > > In an effort to evaluate suspensions earlier so that we can minimize > excess buffering, I've discovered that this is causing problems related > to suspensions that create other suspensions. I think normally this > isn't an issue because we evaluate suspensions at the end, so the nested > suspensions don't actually need to suspend. But now that I'm trying to > evaluate suspensions earlier, the nested suspensions actually do need to > suspend. > > Only about a dozen tests fail right now with my changes to evaluate > suspensions earlier, but the simplest one is > test_sequenceWithComplexType. This test is a sequence of prefix > separated elements: > > https://github.com/apache/incubator-daffodil/blob/master/daffodil-test/src/test/resources/org/apache/daffodil/section14/sequence_groups/sequenceWithComplexType.dfdl.xsd > > I believe what is happening during unparse is we create a suspension to > determine if the unparsed element is zero length or not, which > determines if we should unparse a separator. At some point, the > suspension determines the unparsed element is not zero length, and the > suspension runs the separator unparser and then finishes. > > The problem is that running this separator unparser triggers an > alignment unparser, which at the time of evaluation needs to suspend. > The DOS that this alignment parser suspends on has already been split > (other things have suspended before we reevaluate the separator > suspension), which currently fails due to an assertion. We currently > only ever allow suspending from he last DOS, not a previous DOS. > > My first instinct was to change this assumption to allow suspensions on > an older DOS, and to just sort of stick the new suspension DOS in the > middle of DOS's. I have this working, but this still results in blocks > on the alignment suspensions. I haven't quite figured out what's causing > these blocks. I suspect that there's just something extra that I need to > do to allow arbitrarily inserting DOSs. > > So there's definitely more debugging to do, but wanted to bring this up > before I get too far down the rabbit hole. > > Any thoughts if this is the right approach, or perhaps there's some > other fundamental change to how suspensions need to be handled to allow > this to work? >
