I updated the schema to define the fillByte property at the global level (under xs:annotation). Now the unparsing is working without error. The fillByte property defined for the sequence has no impact whether it is present or not -- it is not overriding the global fillByte value.
Do the code changes you suggest impact the above behavior? On a related note, there is an open ticket that says fillByte allows only raw byte values and not DFDL entities: https://issues.apache.org/jira/browse/DAFFODIL-1646 The example XSD uses DFDL entity value for fillByte. Can DAFFODIL-1646 be closed? Regards, Shashi Ramaka sram...@owlcyberdefense.com Tel: 703-965-3656 -----Original Message----- From: Steve Lawrence <slawre...@apache.org> Sent: Friday, October 16, 2020 1:43 PM To: dev@daffodil.apache.org Subject: Re: DAFFODIL-2377 The issue appears to be with how we don't really require fillByte until runtime. RuntimePropertyMixins.scala defines maybeFillByteEv, which returns the FillByteEv if the dfdl:fillByte property is provided on the schema element. If it's not provided, it's just a Nope and we don't have a fill byte. So our code for creating FillByteEv implies that the dfdl:fillByte property is not mandatory. But then in ProcessorStateBases.scala, the def fillByte function does maybeFillByteEv.get So that requires that maybeFillByteEv is defined when we determine we needed to fill some bytes, and thus the dfdl:fillByteProperty is mandatory. Seems to me we should just make the fill byte property mandatory. I don't know if there are cases where we should actually consider it optional. Solution 1: Change RuntimePropertyMixins.scala so that maybeFillByEv is no longer a Maybe, and it becomes something like this: final lazy val fillByteEv = { val ev = new FillByteEv(fillByte, charsetEv, tci) ev.compile(tunable) ev } So fill byte is always required to be defined in the schema. And then change all references to maybeFillByteEv to just fillByteEv. And follow the variable until ProcessorStateBases.scala just becomes filleByteEv instead of maybeFillByteEv.get This way, if anything ever uses the fillByteEv variable, we require the fillByte property to exist, and if it doesn't we'll get an SDE at schema compile time. And then if fillByte is ever needed, then it will be available. The only issue with this approach is it might break schemas that don't provide fillByte, since we now will always require fill byte where it might not have been technically needed before. But I think we were just getting lucky, and probably most schemas already defined it. Solution 2: Another option would be to just change ProcessorStateBases to be something like this: if (termRuntimeData.maybeFillByteEv.isEmpty) { SDE("fillByte property required") } else { maybeCachedFillByte = MaybeInt(termRuntimeData.maybeFillByteEv.get....) } So at runtime, if we need to fill some bytes but maybeFillByteEv isn't defined (i.e. the schema didn't define fillByte), then we throw a Runtime SDE. This maintains backwards compatibility, but doesn't detect the missing property until runtime, which is unfortunately. I'm not sure if it's wroth maintaining that for this issue though. I'd probably lean towards the first solution and just always require fillByte. On 10/16/20 1:16 PM, Ramaka, Shashi wrote: > I am working on DAFFODIL-2377: Abort instead of diagnostic message. > (https://issues.apache.org/jira/browse/DAFFODIL-2377 > <https://issues.apache.org/jira/browse/DAFFODIL-2377>) > > The attached files can be used to illustrate the bug. > > daffodil parse -s s2377.xsd -o d2377.xml d2377.bin > > daffodil unparse -s s2377.xsd d2377.xml > > Unparsing results in the below error: > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > !! An unexpected exception occurred. This is a bug! !! > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > The schema has three elements. The first element has a length of 1 > byte and alignment of 1 byte while the next two elements have length = > 1 and alignment = 2. The unparser is unparsing the first element > correctly but throwing the above exception while determining the > second element. Specifically it is failing figuring out the fill byte to be > used. > > The unparser is failing on the highlighted line in > ProcessorStateBase.scala in > daffodil-runtime1: > > final def fillByte: Byte = { > if (maybeCachedFillByte.isEmpty) > * maybeCachedFillByte = > MaybeInt(termRuntimeData.maybeFillByteEv.get.evaluate(this).toInt) > * maybeCachedFillByte.get.toByte > } > > Seems like the processor state in termRuntimeData does not have the > fill byte value from the schema. If I update the above fillByte to > return a hard coded byte value, unparse is working fine. > > I'd appreciate any pointers on how to proceed from here. > > Regards, > > Shashi Ramaka > > sram...@owlcyberdefense.com <mailto:sram...@owlcyberdefense.com> > > Tel: 703-965-3656 >
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"; xmlns:fn="http://www.w3.org/2005/xpath-functions"; elementFormDefault="qualified"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/";> <dfdl:format textBidi="no" floating="no" encodingErrorPolicy="replace" leadingSkip="0" alignmentUnits="bytes" alignment="1" trailingSkip="0" textPadKind="none" escapeSchemeRef="" truncateSpecifiedLengthString="no" textTrimKind="none" binaryNumberRep="binary" representation="binary" byteOrder="littleEndian" encoding="ISO-8859-1" sequenceKind="ordered" initiator="" terminator="" separator="" ignoreCase = "yes" initiatedContent="no" fillByte="%NUL;" /> </xs:appinfo> </xs:annotation> <xs:element name="input" dfdl:lengthKind="implicit" dfdl:lengthUnits="bytes" > <xs:complexType> <xs:sequence dfdl:fillByte="%FF;"> <xs:element name="A" type="xs:unsignedInt" dfdl:lengthKind="explicit" dfdl:length="1" dfdl:lengthUnits="bytes" dfdl:alignment="1" dfdl:alignmentUnits="bytes" /> <xs:element name="B" type="xs:unsignedInt" dfdl:lengthKind="explicit" dfdl:length="1" dfdl:lengthUnits="bytes" dfdl:alignment="2" dfdl:alignmentUnits="bytes" /> <xs:element name="C" type="xs:unsignedInt" dfdl:lengthKind="explicit" dfdl:length="1" dfdl:lengthUnits="bytes" dfdl:alignment="2" dfdl:alignmentUnits="bytes" /> </xs:sequence> </xs:complexType> </xs:element> </xs:schema>
