I found the issue.
I noticed that you import the DFDLGeneralFormat.dfdl.xml, but then you don't
use it.
So I checked, and in your list of property definitions in dfdl:format they are
almost all the same as if you had ref="ama:GeneralFormat" except for one
crucial one.
You have separatorSuppressionPolicy="never".
That's not what you want. You want "anyEmpty".
This separatorSuppressionPolicy property is among the most difficult to deal
with in the spec.
The "never" value means "never suppress separators, even if the item is
missing". That is really about fixed length arrays and fixed-size tuples where
the separators are always going to appear even if the element content is
missing.
Think sparse data formats with data like "/1,2,,,5,,7,8/" where the missing
pieces are still delimited by separators.
I think the diagnostic message is not good here, and will create a bug ticket
about it. It should say specifically that it was looking for the separator, not
that the element initiator isn't present. Or it should say it didn't find the
element because it didn't find the initiator, AND also didn't find the
associated separator.
If I remove all your format properties and just use
<dfdl:format ref="ama:GeneralFormat"/> then it works.
One other minor thing: You had textStringPadCharacter="%WSP;" which is a
schema-definition error. I changed it to %SP; which is probably what you
intended, and that's the value in ama:GeneralFormat, so I removed it entirely.
The TDML file that has the test that shows it working is attached.
________________________________
From: Beckerle, Mike <[email protected]>
Sent: Wednesday, February 5, 2020 9:07 AM
To: Patrick Grandjean <[email protected]>; [email protected]
<[email protected]>
Subject: Re: Optional element with initiator
Sorry for duplicate. This time to users list also.
I will try to reproduce the issue and get back to you.
________________________________
From: Patrick Grandjean <[email protected]>
Sent: Tuesday, February 4, 2020 2:31 PM
To: [email protected] <[email protected]>
Subject: Re: Optional element with initiator
Thank for the answer.
Unfortunately, if I try to remove any dfdl:occursCountKind I get an error from
daffodil saying this attribute is missing.
I have removed the postfix attribute, thank you for the tip!
I have set all elements to be optional (minOccurs="0"), but I still get an
Initiator exception:
diff:
No differences
----------------------------------------------------------------- 91
parser: <Initiator/>
bitPosition: 4096
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000200: 3530 3030 3030 3130 3030 3030 3030 3230 5000001000000020
00000210: 3030 4555 5232 2020 2020 2020 2020 3030 00EUR2␣␣␣␣␣␣␣␣00
infoset:
<record>
<it02>
[....]
</it02>
<it03 />
</record>
diff:
No differences
failure:
Parse Error: Initiator '3' not found
Schema context: it03 Location line 71 column 14 in
file:/path/to/apache-daffodil-2.5.0-incubating-bin/bin/../../src/main/resources/grammar.dfdl.xsd
Data location was preceding byte 512
I would expect that if an initiator is not found, the parser would backtrack
and try the next element in the sequence. Is there a policy to be set?
I have tried to replace the initiators with elements having fixed values, but
it seems that the fixed value is not checked. Example below, with fixed="1":
<xs:element name="recordIdentifier" type="xs:string" fixed="1" dfdl:length="1"
dfdl:lengthKind="explicit" />
On Mon, Feb 3, 2020 at 3:17 PM Beckerle, Mike
<[email protected]<mailto:[email protected]>> wrote:
I see a couple issues here.
First your record element is not optional but you have
dfdl:occursCountKind="implicit" on it, suggesting you wanted minOccurs="0" on
it, or maxOccurs="unbounded", but you didn't put that.
Same for it02 element, which probably wants minOccurs="0".
Second, your records type has a sequence with postfix newline separator.
Your inner sequence inside the record type *also* has postfix newline separator.
This means you will have TWO newlines required at the end of your data.
If you don't have two newlines, then it's going to backtrack and, while I can't
construct a specific reason why it ends up giving you an error on it03, I can
imagine that could happen.
I suggest removing the postfix from the inner sequence so it is only infix
separated, and the final newline will come from the outer sequence.
Then I suggest making it02 element optional and record element optional or
removing the dfdl:occursCountKind="implicit" from either or both of them.
Then try again. Post your results here again.
You may want to turn on the trace feature. You get TONS of sometimes obscure
output from trace (fixing that is on our roadmap), but often one can wade
through it and spot where things go wrong.
If your format has many many of these different record sub-types you will
eventually want to switch away from using initiators to a choice using the
dfdl:choiceDispatchKey property.
But we should get what you started with working properly first.
________________________________
From: Patrick Grandjean
<[email protected]<mailto:[email protected]>>
Sent: Monday, February 3, 2020 2:23 PM
To: [email protected]<mailto:[email protected]>
<[email protected]<mailto:[email protected]>>
Subject: Optional element with initiator
Hi!
I'm trying to create my first DFDL schema to parse a fixed column format. The
format may change from one line to the other depending on the first character
of the line. Some lines are optional, therefore I use minOccurs="0" and
dlfl:occursCountKind="implicit". Here is the schema:
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema";
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/";
xmlns:ex="http://example.com";
xmlns:ama="http://www.amadeus.com";
targetNamespace="http://www.amadeus.com";
elementFormDefault="unqualified">
<xs:include
schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
<xs:include schemaLocation="IT01.dfdl.xsd" />
<xs:include schemaLocation="IT02.dfdl.xsd" />
<xs:include schemaLocation="IT03.dfdl.xsd" />
<xs:include schemaLocation="IT05.dfdl.xsd" />
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/";>
<dfdl:format alignment="1"
alignmentUnits="bytes"
encoding="ASCII"
encodingErrorPolicy="replace"
escapeSchemeRef=""
floating="no"
ignoreCase="no"
initiatedContent="no"
initiator=""
leadingSkip="0"
lengthKind="implicit"
lengthUnits="bytes"
outputNewLine="%LF;"
representation="text"
separator=""
separatorSuppressionPolicy="never"
sequenceKind="ordered"
terminator=""
textBidi="no"
textPadKind="none"
textStringJustification="center"
textStringPadCharacter="%WSP;"
textTrimKind="padChar"
trailingSkip="0"
truncateSpecifiedLengthString="no"/>
</xs:appinfo>
</xs:annotation>
<xs:element name="records" type="ama:Records" />
<xs:complexType name="Records">
<xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
<xs:element name="it01" minOccurs="0" maxOccurs="1" type="ama:IT01"
dfdl:initiator="1" dfdl:occursCountKind="implicit" />
<xs:element name="record" type="ama:Record" minOccurs="1"
dfdl:occursCountKind="implicit" />
</xs:sequence>
</xs:complexType>
<xs:complexType name="Record">
<xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
<xs:element name="it02" type="ama:IT02" dfdl:initiator="2"
dfdl:occursCountKind="implicit" />
<xs:element name="it03" minOccurs="0" type="ama:IT03"
dfdl:initiator="3" dfdl:occursCountKind="implicit" />
<xs:element name="it05" minOccurs="0" type="ama:IT05"
dfdl:initiator="5" dfdl:occursCountKind="implicit" />
</xs:sequence>
</xs:complexType>
</xs:schema>
A file would look like this:
1......
2......
5......
Elements it01 (line starting with '1') and it02 (line starting with '2') are
parsed successfully. Element it03 is missing, but since it is optional it is
OK. The problem is that the Daffodil raises an error saying it has not found
initiator '3'. How to make the initiator optional?
Patrick.
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<tdml:testSuite
description="Bug report"
xmlns:tdml="http://www.ibm.com/xmlns/dfdl/testData";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns:xml="http://www.w3.org/XML/1998/namespace";
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/";
xmlns:xs="http://www.w3.org/2001/XMLSchema";
xmlns:gpf="http://www.ibm.com/dfdl/GeneralPurposeFormat";
xmlns:daf="urn:ogf:dfdl:2013:imp:daffodil.apache.org:2018:ext"
xmlns:ex="http://example.com";
xsi:schemaLocation="http://www.ibm.com/xmlns/dfdl/testData tdml.xsd"
defaultRoundTrip="onePass"
xmlns:ama="http://example.com";
>
<tdml:defineSchema name="s1" elementFormDefault="unqualified">
<xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
<!--
In format below. Remove separatorSuppressionPolicy, or change
value to "never" to reproduce the problem. You will get an error
about initiator "3" not found.
separatorSuppressionPolicy="never" means the separators will be present.
It is generally for fixed-length arrays, or sequences that are fixed-length
tuples of elements.
-->
<dfdl:format
ref="ex:GeneralFormat"
separatorSuppressionPolicy="anyEmpty"
/>
<xs:element name="records" type="ama:Records" />
<xs:complexType name="Records">
<xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
<xs:element name="it01" minOccurs="0" type="ama:IT01" dfdl:initiator="1" />
<xs:element name="record" type="ama:Record" />
</xs:sequence>
</xs:complexType>
<xs:complexType name="Record">
<xs:sequence dfdl:separator="%NL;">
<xs:element name="it02" type="ama:IT02" dfdl:initiator="2" />
<xs:element name="it03" minOccurs="0" type="ama:IT03" dfdl:initiator="3" />
<xs:element name="it05" minOccurs="0" type="ama:IT05" dfdl:initiator="5" />
</xs:sequence>
</xs:complexType>
<xs:simpleType name="delimString" dfdl:lengthKind="delimited">
<xs:restriction base="xs:string" />
</xs:simpleType>
<xs:simpleType name="IT01">
<xs:restriction base="ama:delimString" />
</xs:simpleType>
<xs:simpleType name="IT02">
<xs:restriction base="ama:delimString" />
</xs:simpleType>
<xs:simpleType name="IT03">
<xs:restriction base="ama:delimString" />
</xs:simpleType>
<xs:simpleType name="IT04">
<xs:restriction base="ama:delimString" />
</xs:simpleType>
<xs:simpleType name="IT05">
<xs:restriction base="ama:delimString" />
</xs:simpleType>
</tdml:defineSchema>
<tdml:parserTestCase name="test1" model="s1">
<tdml:document><![CDATA[1aaa
2bbb
5eee
]]></tdml:document>
<tdml:infoset>
<tdml:dfdlInfoset>
<ama:records>
<it01>aaa</it01>
<record>
<it02>bbb</it02>
<it05>eee</it05>
</record>
</ama:records>
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
</tdml:testSuite>
