As Brandon points out, this might give issues to non-byte size
encodings? I assume mandatory text alignment applies for these as well,
which I think deals with such issues. And I assume MTA still applies
even with raw byte entities.

Seems like the algorithm is something like:

  1) Set a mark
  2) Apply delimiter MTA
  3) Check if delimiter bits match current bits
  4) If not match, decode a single char in element encoding and record
  5) Repeat from 1

Re: escape characters, I think escape characters are always in the
encoding of the element of which they are applied since escapeSchemes do
not include a dfdl:encoding property?

Requires extra logic in the above, maybe even attempting the decode
before checking delimiters since escape char effectively disables
delimiter scanning, but I don't think anything too crazy.

I wonder if some of the DFA complexity goes away?



On 4/30/20 2:47 PM, Beckerle, Mike wrote:
> 
> The encoding for the delimiter is the encoding in effect on the schema 
> component carrying the property. Making them take on contextual encodings 
> makes things much too complicated.
> 
> So yeah, I think in your case, if we're scanning for that "§" but we're using 
> a decoder for ASCII, that's incorrect.
> 
> These mixed encoding cases are all corner cases anyway, so they don't have to 
> be natural or easy. The rules simply have to be easy to interpret.
> 
> So your root element defines a terminator.
> 
> That terminator's encoding has *nothing* to do with the encoding specified 
> for a contained element within root. It is not that contained element's 
> terminator, it is the root's terminator.
> 
> The semantics of delimiter scanning in DFDL is in fact something that 
> requires lowering the delimiters to byte patterns. This is required based on 
> mixed scenarios like this, but also based on features like Byte-Value 
> entities e.g., %#rHH; which specifies a hex byte that can appear, even in the 
> middle of characters, when that byte makes no sense.
> 
> <element name="foo" type="xs:int" dfdl:terminator="11%#r88;99" 
> dfdl:encoding="utf-16BE"/>
> 
> So the terminator of the above is bytes 0031 0031 88 0039 0039.  See how that 
> 88 is just thrown in there. Makes no sense in ANY encoding. We're even 
> screwing up the character alignment here.
> 
> That means scanning for delimiters in DFDL requires us to lower the scanning 
> to bytes.
> 
> Of course Daffodil doesn't implement %#rHH; byte-value (aka raw-bytes) 
> entities except for one special case which is specifying the fill byte 
> property.  And our scanning is currently character oriented.
> 
> So, what does it mean to scan for say a UTF-16 character '1' as terminator of 
> an element that is in say, ASCII ?
> 
> It means you are searching through the bytes ignoring ASCII, decoding it as 
> UTF-16, looking for '1' (which is bytes 00 31).  Then having found a 0031, 
> the preceding bytes are then decoded as ASCII.
> 
> Pretty sure Daffodil scanning isn't doing that.
> 
> 
> ________________________________
> From: Sloane, Brandon <bslo...@tresys.com>
> Sent: Thursday, April 30, 2020 12:01 PM
> To: dev@daffodil.apache.org <dev@daffodil.apache.org>
> Subject: Re: Incorrect delimiter scanning when mixed encodings?
> 
> Without looking at the spec, I would expect that delimiters be defined by the 
> encoding the the element that defines the delimeter; so Daffodil is buggy in 
> the case you describe. However, there are a couple of complications we have 
> to consider:
> 
> 1) What if instead of a terminator, we had a separator; and the separator is 
> a valid character in both encoding; but has a different bytecode
> 
>  <xs:element name="root" >
>        <xs:complexType>
>          <xs:sequence dfdl:separator=",">
>            <xs:element name="name" type="xs:string" maxOccurs="2" 
> encoding="FOO"/>
>            <xs:element name="address" type="xs:string" maxOccurs="2" 
> encoding="BAR"/>
>          </xs:sequence>
>        </xs:complexType>
>     </xs:element>
> 
> In this case, I would expect the separator to be interperated based on the 
> encoding of the individual elements, which is obviously not consistent with 
> my expectation from your example. There is also the instance of the separator 
> occurring between the two element types. So even in this case my naive 
> expectation is not consistent.
> The correct answer here is probably to say that this example schema is wrong, 
> and there should be 2 sequences, each defining their own separator.
> 
> 2) What if the encodings have a different alignment? For instance, if the 
> outer encoding that defines the delimiter is 8-bit and byte alligned, with a 
> 7-bit inner encoding, should we look forward to the next byte boundary after 
> every 7 bit character?
> 
> 3) How does this interact with escape sequences?
> 
> The solution here might be to think through some restrictions on where 
> encoding changes are allowed to occur. I am not sure it is possible to give 
> reasonable semantics for everything over a region that spans multiple 
> encodings.
> ________________________________
> From: Steve Lawrence <slawre...@apache.org>
> Sent: Thursday, April 30, 2020 11:15 AM
> To: dev@daffodil.apache.org <dev@daffodil.apache.org>
> Subject: Incorrect delimiter scanning when mixed encodings?
> 
> Say we have a schema like this:
> 
>   <xs:schema
>     xmlns:xs="http://www.w3.org/2001/XMLSchema";
>     xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/";>
> 
>     <xs:include
> schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
> 
>     <xs:annotation>
>       <xs:appinfo source="http://www.ogf.org/dfdl/";>
>         <dfdl:format ref="GeneralFormat" lengthKind="delimited"
>           encoding="ISO-8859-1" />
>       </xs:appinfo>
>     </xs:annotation>
> 
>      <xs:element name="root" dfdl:terminator="§">
>        <xs:complexType>
>          <xs:sequence>
>            <xs:element name="name" type="xs:string" />
>          </xs:sequence>
>        </xs:complexType>
>     </xs:element>
> 
>   </xs:schema>
> 
> So we have a format that is all ISO-8859-1, and a delimited string
> called "name", and the root is terminated by "§" in the ISO-8859-1
> encoding. If we have data that looks like this:
> 
>   text§
> 
> It will parse to this:
> 
>   <root>
>     <name>text</name>
>   </root>
> 
> Now say we want just the "name" element to have a different encoding, so
> we change it to this:
> 
>   <xs:element name="name" type="xs:string" dfdl:encoding="US-ASCII" />
> 
> Now the terminator defined on the root element is in a different
> encoding than the delimited element. Note that the terminator § isn't
> even valid in this encoding.
> 
> Currently, Daffodil does not successfully parse this. It scans the data,
> decoding a single character at a time looking for a delimiter.
> Eventually it gets to the § ata, the decoder says it's not valid in our
> ASCII encoding and converts it to the unicode replacement character.
> This of course doesn't match the delimiter we're looking for and
> continue on. The delimeter scanner then hits the end of data, and errors
> when it never finds the root termiantor.
> 
> Is this the correct behavior, or is our delimiter scanning fundamentally
> broken?
> 
> I wonder if the correct behavior is when the terminator comes into scope
> we should immediately encode it into its bytes. Delimiter scanning only
> looks for these bytes and doesn't actually decode any data. And only
> when bytes that match a delimiter are found do we decode all the bytes
> up until that point?
> 
> Is this reasonable, or is this type of thing just not allowed?
> 
> 
> Note that this is somewhat hypothetical. I don't know of any formats
> that mix encodings like this, but this popped into my head looking at
> DAFFODIL-2323 which complains about encoding property and a sequence,
> which can have a siilar issue if the sequence encoding differs from
> element encodings.
> 

Reply via email to