Re: Please review mock up idea for checksum calculations in DFDL

[Steve Lawrence]

On 8/23/21 1:51 PM, Beckerle, Mike wrote:
> From: Steve Lawrence 
> Sent: Monday, August 9, 2021 12:18 PM
> To: dev@daffodil.apache.org 
> Subject: Re: Please review mock up idea for checksum calculations in DFDL
> 

--- snip ---

> 
> 2) For the IPv4 layer, it feels a bit unfortunate to have to split the
> CRC into two separate layers, since the CRC algorithm is really just a
> checksum over the whole header with just the checksum field treated as
> if it were zero. Is it possible to have a property that just specifies
> that the Nth byte doesn't contribute? Maybe something like:
> 
>dfdlx:runtimeProperties="ignoreByte=5">...
> 
> @@@ In the case of the IPv4 checksum, it can just hardcode the fact
that it skips those specific bytes.  I included the splitting into two
separate layers just to illustrate that this complexity could be
handled. I will look at recasting this as just one checksum layer and
see how it comes out. I think the other example of the GPS data format
with parity bit computations, is worth looking at as that one is fairly
complicated in which bits contribute in what ways.

Thinking more about this, I'm wondering if this is even possible to have
a checksum field inside the checksum layer, as I suggested? I *think*
that would cause circularities during unparse?

Say we have this schema, which is a simplified version of IPv4:

  



  

So we have multiple fields that are all checksumed, where one of the
fields (field2 in this case) actually stores the checksum. And the bytes
associated with field are just skipped during the checksum calculation.

First field1 is unparsed. This goes to some InputStream, which the
checksum layer can start reading from and calculating the checksum. All
good so far.

Then field2 is unparsed. But because it is an OVC element, we create a
buffer for the eventual data, write nothing, and suspend until the
$checksum variable is set. All normal so far.

Then field3 is unparsed. But because the previous field is buffered,
this too must be buffered. We can still unparse data to this buffer, but
because it's being buffer, nothing is written to the InputStream that
the checksum layer is reading from.

And know we're in a deadlock. field2 is suspended waiting for $checksum
to be set. But we can't deliver any of these buffers to the underlying
InputStream so the checksum layer can finish it's calculation. Which
means $checksum is never set. So field2 can't unsuspended, etc. We're in
a loop.



A potential workaround might be to have special logic where the field3
buffer can be written to the checksum layer (since field two doesn't
matter in the calculation). And the checksum layer just knows field2 was
skipped. This would then allow the checksum layer to finish, and thus
field2 to be unparsed. But then the checksum layer needs to also keep a
buffer so that it can insert the unparsed field2 OVC value before the
field3 data. This seems pretty specialized though. And doesn't take into
account things like potential alignment that might not even be known
until field2 is actually unparsed, which would change the checksum value.

So I think we do need to use the approach where we split the checksum
into two different layers and combine them.

Re: Please review mock up idea for checksum calculations in DFDL

[Steve Lawrence]

On 8/25/21 1:41 PM, Beckerle, Mike wrote:
> One further comment at the end.
> 
> 
> From: Steve Lawrence 
> Sent: Monday, August 23, 2021 2:23 PM
> To: dev@daffodil.apache.org 
> Subject: Re: Please review mock up idea for checksum calculations in DFDL
> 
> On 8/23/21 1:51 PM, Beckerle, Mike wrote:
>> Comments below see @@@mb
>>
>> 
>> From: Steve Lawrence 
>> Sent: Monday, August 9, 2021 12:18 PM
>> To: dev@daffodil.apache.org 
>> Subject: Re: Please review mock up idea for checksum calculations in DFDL
>>
>> Some comments:
>>
>> 1) I like the idea that the layers write to a variable, but it seems
>> like the variables are hard coded in the layer transformer? What are
>> your thoughts on having the variable defined in a property so that the
>> user has more control over the naming/definition of it, maybe via
>> something like dfdlx:runtimeProperties? For example:
>>
>>   > dfdlx:runtimeProperties="resultVariable=checksumPart1">...
>>
>> @@@ given that a layer transform can be defined with a unique namespace 
>> defined by way of a URI, there's never a need to be
>> concerned about naming conflicts. So I think ability to choose the variables 
>> names and provide them is overkill.
> 
> This is maybe a bit contrived, but one benefit of some configurability
> is that if you have a format with two of the same checksums for
> different parts of the data, you don't need newVariableInstance stuff.
> For example:
> 
>   
>   
> 
>   
>  dfdl:layerParameters="res=checksumHeader">
>   
> 
>  dfdl:layerParameters="res=checksumPayload">
>   
> 
>   
> 
> So it's just a bit cleaner looking. Though, I'm not sure that's a strong
> argument for configuring the variables. I imagine in most formats where
> there's multiple of the same checksums then it's in an array and you'd
> need new variable instance since the number of checksums isn't known.
> 
> I think this is a "let's see" kind of issue. We can use hardwired variables 
> for now, and add a feature later to pass in QNames of variables for the layer 
> to use if we find it too clumsy.

Good point. Keep it easy at first make sense. It should be easy to add a
feature to override the hardwired name if we realize it's needed.

Re: Please review mock up idea for checksum calculations in DFDL

[Beckerle, Mike]

One further comment at the end.


From: Steve Lawrence 
Sent: Monday, August 23, 2021 2:23 PM
To: dev@daffodil.apache.org 
Subject: Re: Please review mock up idea for checksum calculations in DFDL

On 8/23/21 1:51 PM, Beckerle, Mike wrote:
> Comments below see @@@mb
>
> 
> From: Steve Lawrence 
> Sent: Monday, August 9, 2021 12:18 PM
> To: dev@daffodil.apache.org 
> Subject: Re: Please review mock up idea for checksum calculations in DFDL
>
> Some comments:
>
> 1) I like the idea that the layers write to a variable, but it seems
> like the variables are hard coded in the layer transformer? What are
> your thoughts on having the variable defined in a property so that the
> user has more control over the naming/definition of it, maybe via
> something like dfdlx:runtimeProperties? For example:
>
>dfdlx:runtimeProperties="resultVariable=checksumPart1">...
>
> @@@ given that a layer transform can be defined with a unique namespace 
> defined by way of a URI, there's never a need to be
> concerned about naming conflicts. So I think ability to choose the variables 
> names and provide them is overkill.

This is maybe a bit contrived, but one benefit of some configurability
is that if you have a format with two of the same checksums for
different parts of the data, you don't need newVariableInstance stuff.
For example:

  
  

  

  


  

  

So it's just a bit cleaner looking. Though, I'm not sure that's a strong
argument for configuring the variables. I imagine in most formats where
there's multiple of the same checksums then it's in an array and you'd
need new variable instance since the number of checksums isn't known.

I think this is a "let's see" kind of issue. We can use hardwired variables for 
now, and add a feature later to pass in QNames of variables for the layer to 
use if we find it too clumsy.

...

Re: Please review mock up idea for checksum calculations in DFDL

[Steve Lawrence]

On 8/23/21 1:51 PM, Beckerle, Mike wrote:
> Comments below see @@@mb
> 
> 
> From: Steve Lawrence 
> Sent: Monday, August 9, 2021 12:18 PM
> To: dev@daffodil.apache.org 
> Subject: Re: Please review mock up idea for checksum calculations in DFDL
> 
> Some comments:
> 
> 1) I like the idea that the layers write to a variable, but it seems
> like the variables are hard coded in the layer transformer? What are
> your thoughts on having the variable defined in a property so that the
> user has more control over the naming/definition of it, maybe via
> something like dfdlx:runtimeProperties? For example:
> 
>dfdlx:runtimeProperties="resultVariable=checksumPart1">...
> 
> @@@ given that a layer transform can be defined with a unique namespace 
> defined by way of a URI, there's never a need to be
> concerned about naming conflicts. So I think ability to choose the variables 
> names and provide them is overkill.

This is maybe a bit contrived, but one benefit of some configurability
is that if you have a format with two of the same checksums for
different parts of the data, you don't need newVariableInstance stuff.
For example:

  
  

  

  


  

  

So it's just a bit cleaner looking. Though, I'm not sure that's a strong
argument for configuring the variables. I imagine in most formats where
there's multiple of the same checksums then it's in an array and you'd
need new variable instance since the number of checksums isn't known.


> I think of the variable definitions as coming from an imported schema that 
> one must have to use the layer transform.
> Right now we don't have a way of declaring a layer transform when defined 
> outside of the daffodil code base in a pluggable fashion, but assume we had 
> something like  className="com.myco.foobar"/> which would
> also appear in that import file, then accessing and using the layer transform 
> and its associated variables would all be obtained from the one import 
> statement.
> 
> 2) For the IPv4 layer, it feels a bit unfortunate to have to split the
> CRC into two separate layers, since the CRC algorithm is really just a
> checksum over the whole header with just the checksum field treated as
> if it were zero. Is it possible to have a property that just specifies
> that the Nth byte doesn't contribute? Maybe something like:
> 
>dfdlx:runtimeProperties="ignoreByte=5">...
> 
> @@@ In the case of the IPv4 checksum, it can just hardcode the fact that it 
> skips those specific bytes.  I included the splitting into two separate 
> layers just to illustrate that this complexity could be handled. I will look 
> at recasting this as just one checksum layer and see how it comes out. I 
> think the other example of the GPS data format with parity bit computations, 
> is worth looking at as that one is fairly complicated in which bits 
> contribute in what ways.

Agreed. I wasn't sure if there the IPv4 checks is specific to IPv4 or if
there are other uses where different bytes (or no bytes) are ignored.
Just thinking about re usability, but that's maybe more of an
implementation detail.

I'll take a look at the GPS example.

> 3) As for implementing the checksums, have you put any thought into
> making that extensible? For example, I'm wondering if we only have a
> single "checksum" layer, and then the dfdlx:runtimeProperties determines
> which algorithm to use? E.g.
> 
>dfdlx:runtimeProperties="algorithm=crc32">...
> 
>dfdlx:runtimeProperties="algorithm=ipv4header">...
> 
> And then people can register different checksum algorithms without
> having to reimplement their own layer? Or maybe we keep it simple and
> the default checksum layer just supports a handful of the most common
> checksums (maybe those supported by some preexisting checksum library?)
> 
> People could still implement their own pluggable checksum layer if they
> need something we don't support, but this would cover the most common
> cases and avoids a proliferation of a bunch of different layers that are
> basically the same except for some minor algorithm details.
> 
> @@@ This refactoring can of course be done. But isn't needed to get started. 
> Parameters to transform algorithms can be passed in variables, or could be 
> specified using an extensible property bag such as dfdlx:runtimeProperties as 
> you have shown. We may want a dedicated dfdl:layerParameters property since 
> we have other layering-specific properties (e.g., for layering length kind, 
> etc.) rather than using a generic hook. Ideally layering transformers could 
> check these 

Re: Please review mock up idea for checksum calculations in DFDL

[Beckerle, Mike]

Comments below see @@@mb


From: Steve Lawrence 
Sent: Monday, August 9, 2021 12:18 PM
To: dev@daffodil.apache.org 
Subject: Re: Please review mock up idea for checksum calculations in DFDL

Some comments:

1) I like the idea that the layers write to a variable, but it seems
like the variables are hard coded in the layer transformer? What are
your thoughts on having the variable defined in a property so that the
user has more control over the naming/definition of it, maybe via
something like dfdlx:runtimeProperties? For example:

  ...

@@@ given that a layer transform can be defined with a unique namespace defined 
by way of a URI, there's never a need to be
concerned about naming conflicts. So I think ability to choose the variables 
names and provide them is overkill.

I think of the variable definitions as coming from an imported schema that one 
must have to use the layer transform.
Right now we don't have a way of declaring a layer transform when defined 
outside of the daffodil code base in a pluggable fashion, but assume we had 
something like  
which would
also appear in that import file, then accessing and using the layer transform 
and its associated variables would all be obtained from the one import 
statement.

2) For the IPv4 layer, it feels a bit unfortunate to have to split the
CRC into two separate layers, since the CRC algorithm is really just a
checksum over the whole header with just the checksum field treated as
if it were zero. Is it possible to have a property that just specifies
that the Nth byte doesn't contribute? Maybe something like:

  ...

@@@ In the case of the IPv4 checksum, it can just hardcode the fact that it 
skips those specific bytes.  I included the splitting into two separate layers 
just to illustrate that this complexity could be handled. I will look at 
recasting this as just one checksum layer and see how it comes out. I think the 
other example of the GPS data format with parity bit computations, is worth 
looking at as that one is fairly complicated in which bits contribute in what 
ways.

3) As for implementing the checksums, have you put any thought into
making that extensible? For example, I'm wondering if we only have a
single "checksum" layer, and then the dfdlx:runtimeProperties determines
which algorithm to use? E.g.

  ...

  ...

And then people can register different checksum algorithms without
having to reimplement their own layer? Or maybe we keep it simple and
the default checksum layer just supports a handful of the most common
checksums (maybe those supported by some preexisting checksum library?)

People could still implement their own pluggable checksum layer if they
need something we don't support, but this would cover the most common
cases and avoids a proliferation of a bunch of different layers that are
basically the same except for some minor algorithm details.

@@@ This refactoring can of course be done. But isn't needed to get started. 
Parameters to transform algorithms can be passed in variables, or could be 
specified using an extensible property bag such as dfdlx:runtimeProperties as 
you have shown. We may want a dedicated dfdl:layerParameters property since we 
have other layering-specific properties (e.g., for layering length kind, etc.) 
rather than using a generic hook. Ideally layering transformers could check 
these properties statically and issue SDEs if misused.


On 7/30/21 2:29 PM, Beckerle, Mike wrote:
> I would like comments on the layering enhancement to enable checksum
> computations in DFDL schemas.
>
>
> This is a high-priority feature for Daffodil's next release 3.2.0, especially
> for cybersecurity applications of Daffodil, which I know a number of us are
> involved in.
>
>
> I've produced a mock-up of how it would look, with lots of annotations in a 
> WIP
> pull request on the ethernetIP DFDL schema. I only did the mock-up for the 
> IPV4
> element, so look at that element in the ethernetIP.dfdl.xsd.
>
> (UDP and TCP packets have their own additional checksums - I didn't mock up
> those, just IPV4)
>
>
> This is at https://github.com/DFDLSchemas/ethernetIP/pull/1
> <https://github.com/DFDLSchemas/ethernetIP/pull/1>
>
>
> This doesn't run, it's just an initial mock-up of the ideas for
> checksum/CRC/parity recomputation capability as a further simple extension of
> the existing DFDL layering extension.
>
>
> The layering extension itself is described here:
>
> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Data+Layering+for+Base64%2C+Line-Folding%2C+Compression%2C+Etc
> <https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Data+Layering+for+Base64%2C+Line-Folding%2C+Compression%2C+Etc>
>
>
> I did notice that none of the published DFDLSchemas actually use the layerin

Re: Please review mock up idea for checksum calculations in DFDL

[Steve Lawrence]

Some comments:

1) I like the idea that the layers write to a variable, but it seems
like the variables are hard coded in the layer transformer? What are
your thoughts on having the variable defined in a property so that the
user has more control over the naming/definition of it, maybe via
something like dfdlx:runtimeProperties? For example:

  ...

2) For the IPv4 layer, it feels a bit unfortunate to have to split the
CRC into two separate layers, since the CRC algorithm is really just a
checksum over the whole header with just the checksum field treated as
if it were zero. Is it possible to have a property that just specifies
that the Nth byte doesn't contribute? Maybe something like:

  ...


3) As for implementing the checksums, have you put any thought into
making that extensible? For example, I'm wondering if we only have a
single "checksum" layer, and then the dfdlx:runtimeProperties determines
which algorithm to use? E.g.

  ...

  ...

And then people can register different checksum algorithms without
having to reimplement their own layer? Or maybe we keep it simple and
the default checksum layer just supports a handful of the most common
checksums (maybe those supported by some preexisting checksum library?)

People could still implement their own pluggable checksum layer if they
need something we don't support, but this would cover the most common
cases and avoids a proliferation of a bunch of different layers that are
basically the same except for some minor algorithm details.


On 7/30/21 2:29 PM, Beckerle, Mike wrote:
> I would like comments on the layering enhancement to enable checksum 
> computations in DFDL schemas.
> 
> 
> This is a high-priority feature for Daffodil's next release 3.2.0, especially 
> for cybersecurity applications of Daffodil, which I know a number of us are 
> involved in.
> 
> 
> I've produced a mock-up of how it would look, with lots of annotations in a 
> WIP 
> pull request on the ethernetIP DFDL schema. I only did the mock-up for the 
> IPV4 
> element, so look at that element in the ethernetIP.dfdl.xsd.
> 
> (UDP and TCP packets have their own additional checksums - I didn't mock up 
> those, just IPV4)
> 
> 
> This is at https://github.com/DFDLSchemas/ethernetIP/pull/1 
> 
> 
> 
> This doesn't run, it's just an initial mock-up of the ideas for 
> checksum/CRC/parity recomputation capability as a further simple extension of 
> the existing DFDL layering extension.
> 
> 
> The layering extension itself is described here:
> 
> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Data+Layering+for+Base64%2C+Line-Folding%2C+Compression%2C+Etc
>  
> 
> 
> 
> I did notice that none of the published DFDLSchemas actually use the layering 
> transforms that we've built into Daffodil. There are some non-public DFDL 
> schemas that do use this extension to do line-folding transformations.
> 
> 
> There are, however, tests showing the DFDL layering extension in daffodil's 
> code 
> base. See
> 
> https://github.com/apache/daffodil/blob/master/daffodil-test/src/test/resources/org/apache/daffodil/layers/layers.tdml
>  
> 
> and search for dfdlx:layerTransform property.
> 
> 
> The mock-up effectively proposes allowing layer transforms to read and write 
> DFDL variables, as a means of them accepting input parameters, and as the 
> means 
> of them computing and returning output results.
> 
> 
> I plan to do a couple other mock-ups of a check-digit calculation, and some 
> parity bit computations, but this IPV4 is enough to get the gist of the idea.
> 
> 
> I'd appreciate feedback on this, which you can do on the pull request in the 
> usual github code review manner.
> 
> 
> -mikeb
> 
> 
> 
> 
> Mike Beckerle | Principal Engineer
> 
> mbecke...@owlcyberdefense.com 
> 
> P +1-781-330-0412
>