So this issue of when DSOM is "done" is key. DSOM is done when all the objects
that are needed to resolve DFDL annotations have been constructed.
I would say this is when all DSOM objects corresponding to "pieces of XML" in
the schema exist, and when the formatAnnotation for each
AnnotatedSchemaComponent exists. At that point you can reference properties
and other annotations like statements all you want, and SDEs won't end up in
circular stack overflows due to the need for a DSOM object.
Note that top level decls/defs might not have any DSOM object created for them
if they are not referenced.
But this is ALL before anything invokes any method/member of any of the Grammar
mixins. So the code you cite below is (or should be) after the DSOM graph is
fully constructed.
It is exactly this use of dfdl:sequenceKind to decide what DSOM object to
create that is, I believe, a mistake in DSOM. The selection of either a
Sequence or UnorderedSequence DSOM object in the DSOM graph is depending on
dfdl:sequenceKind, which is a scoped DFDL property, and scope resolution
depends on the DSOM graph. Resolution of the dfdl:sequenceKind method has to
start from the correct start of the chain.
Consider this:
---File 1----
<dfdl:format sequenceKind="unordered".../>
... elsewhere in that file....
<group ref="mySeqGr"/>
----File 2----
<group name="mySeqGr">
<sequence> <!-- will be ordered or not depending on group refs -->
...
</sequence>
</group>
So to determine ordered or unordered, we have to look back at the group
reference and start the scan for properties there. This property might even
come from the default format of the schema file containing the group reference,
so we really need the proper DSOM structures in place before we ask for the
dfdl:sequenceKind.
I bet this wouldn't work today because Daffodil would miss the
dfdl:sequenceKind found by way of the group ref. Maybe it would work, but I
wouldn't bet on it.
The right design is to construct a Sequence every time, and construct the
children DSOM objects of the Sequence every time. We have the UnorderedSequence
object become, if needed, just a helper object used by the Sequence. Honestly a
helper object may not be needed. The Sequence method code can just evaluate
some lazy vals to decide whether to put down SeqComp or
UnorderedSequenceCombinator, and compute the parameters they need from the DSOM
objects for the contained children.
One way to do this with some uniformity is to construct a Sequence every time
as part of the DSOM tree, and then construct the DSOM objects for that
Sequence's children. After that and when DSOM is fully constructed, then when
that Sequence is asked for some member/method then if that Sequence determines
it is unordered, to then construct a UnorderedSequence, (passing in the DSOM
objects for the children to the constructor). The original Sequence would then
delegate everything to the UnorderedSequence object. The consistent thing to do
would then be to have dsom.Sequence construct either an OrderedSequence or an
UnorderedSequence, both of which implement the same interface. The Sequence
then decides which one to create, and delegates everything to that object.
It's using a delegation pattern instead of an inheritance pattern because we
want to decide which class to create later, when we have more information.
In that case, the OrderedSequence and UnorderedSequence objects might be DSOM
objects out of convenience. But they're not ordinary DSOM objects because they
do not recursively parse their own children into DSOM objects. The original
Sequence object would be a DSOM object, but really would just be a thin proxy
delegating everything except child construction to the other objects.
________________________________
From: Taylor Wise
Sent: Friday, November 10, 2017 10:07:25 AM
To: Mike Beckerle; [email protected]
Subject: Re: Stack overflows, UnorderedSequences, DSOM
Also, at what point is it safe to construct these helper objects? When is DSOM
done?
For instance, if you go to SequenceGrammarMixin you'll see that we have a bunch
of lazy evaluations concerning orderedSequenceContent and
unorderedSequenceContent.
lazy val unorderedSequenceContent = prod("unorderedSequenceContent") {
lazy val uoseq = self.unorderedSeq.get
UnorderedSequenceCombinator(this, uoseq.terms)
}
Should the above delay the reference to self.unorderedSeq until it's inside the
UnorderedSequenceCombinator? This would delay reference to SequenceKind and
any construction of the UnorderedSequence object until the 'parser' of
UnorderedSequenceCombinator is referenced/used.
Actually, this appears to be an issue for the grammar as well.
SequenceGrammarMixin references SequenceKind in order to determine which
'content' to use. Does this suggest then that really the UnorderedSequence
should use the SequenceCombinator? The only thing that differs would be the
'body' of the object we pass to the underlying parser. And at the time that
the Combinator is used, is only when the object needs to exist.
________________________________
From: Mike Beckerle
Sent: Friday, November 10, 2017 8:21:03 AM
To: [email protected]; Taylor Wise
Subject: Stack overflows, UnorderedSequences, DSOM
So I've had some stack-overflows while debugging of late.
This has lead to thinking about the reasons why. There are some principles of
operation for the code-base of Daffodil that are not articulated, but that need
to be followed to stay out of trouble.
I am planning to add package.scala files describing these to the code base.
For DSOM one is this:
* DSOM components should be selected and constructed without reference to
DFDL annotations, (including DFDL properties) carried on any schema component.
This avoids a chicken/egg situation. The DSOM graph is needed in order to
implement DFDL's property scoping rules; so you can't invoke those mechanisms
in order to build the DSOM graph itself, or you will very likely end up with a
circularity/stack-overflow.
You can; however, look at the XML and its attributes. E.g., the XSD ref
property, which is a QName for a global element/type/group. There are also some
DFDL non-properties - cannot be put into scope - that are needed at the time
DSOM objects are being constructed. One example is the dfdl:hiddenGroupRef
"property" on xs:sequence XML elements. In general it is acceptable to look at,
and even resolve global QName references at DSOM graph construction time.
Coming back to the issue of unordered sequences...
The above suggests a design flaw in DSOM currently, which is the
UnorderedSequence class. In order to decide whether to construct this class or
just a regular Sequence class instance, you have to examine the
dfdl:sequenceKind property, which can be scoped.
Hence, I claim, the UnorderedSequence class should not exist in current form,
but an isUnordered attribute on Sequence should provide the information about
whether the class is ordered or not, and that is determined only *after* the
DSOM graph is constructed. Once the DSOM Sequence class has been constructed,
then regular (usually lazy val) class members on Sequence can construct any
helper objects that are needed for dealing with the complexity of compilation
for unordered sequences.
Comments?
