Yup. This is a real situation. I think this happens in fixed length data for
very mundane reasons explained in the attached slides.
Practically speaking, in Link16 this comes up like twice or 3 times, so this
isn't worth enhancing DFDL/Daffodil for unless this phenomenon is observed in
more places. I have only seen it in Link16, though per the attached slides the
problem could occur a lot in fixed length legacy data formats.
The workaround is just a choice, where you have a "late discriminator" on the
tag, when you finally get to it.
________________________________
From: Sloane, Brandon <bslo...@tresys.com>
Sent: Wednesday, May 29, 2019 1:00:04 PM
To: dev@daffodil.apache.org
Subject: Proposal: Add support for lookahead in DFDL expressions
In developing schema for link16, I have encountered a situation that I do not
believe Daffodil currently has a good solutions for. It is a tagged union,
where the tag comes after the union.
In theory, the schema would look like:
<xs:choice dfdl:choiceDispatchKey="{ tag }">
<xs:element name="a" type="xs:int" dfdl:length="16" dfdl:choiceBranchKey="1"/>
<xs:element name="b" type="xs:int" dfdl:length="16" dfdl:choiceBranchKey="2"/>
</xs:choice>
<xs:element name="tag" type="xs:int" dfdl:length="8" />
Obviously, this schema doe not work because it requires look-ahead. In general
it is not possible to make this sort of schema work, because in order to
determine where in the bitstream <tag> is, one would first need to know the
length of the choice, which cannot (generally speaking) be determined before
parsing completes.
However, in this case, the lookahead is possible in principle, because the
choice happens to be fixed length (as it should be in any sane format where the
tag follows the union).
I believe that we can support this usecase with a much less invasive mechanism
than infoset lookahead. In particular, we can support this with bytestream
lookahead in the DFDL expressions, as below:
<xs:choice dfdl:choiceDispatchKey="{ dfdl:lookAhead(16,8) }">
<xs:element name="a" type="xs:int" dfdl:length="16" dfdl:choiceBranchKey="1"/>
<xs:element name="b" type="xs:int" dfdl:length="16" dfdl:choiceBranchKey="2"/>
</xs:choice>
<xs:element name="tag" type="xs:int" dfdl:length="8" />
The dfdl:lookAhead function takes as input a relative offset, o, and length, n,
and returns the n bits located o bits passed the current location, interperated
as an unsigned integer.
>From am implementation standpoint, there should be no difficulty in adding
>this, as the parser need only peek into the buffer it already has.
Brandon T. Sloane
Associate, Services
bslo...@tresys.com | tresys.com
