Re: Please review mock up idea for checksum calculations in DFDL

[Steve Lawrence]

On 8/23/21 1:51 PM, Beckerle, Mike wrote:
> Comments below see @@@mb
> 
> 
> From: Steve Lawrence 
> Sent: Monday, August 9, 2021 12:18 PM
> To: dev@daffodil.apache.org 
> Subject: Re: Please review mock up idea for checksum calculations in DFDL
> 
> Some comments:
> 
> 1) I like the idea that the layers write to a variable, but it seems
> like the variables are hard coded in the layer transformer? What are
> your thoughts on having the variable defined in a property so that the
> user has more control over the naming/definition of it, maybe via
> something like dfdlx:runtimeProperties? For example:
> 
>dfdlx:runtimeProperties="resultVariable=checksumPart1">...
> 
> @@@ given that a layer transform can be defined with a unique namespace 
> defined by way of a URI, there's never a need to be
> concerned about naming conflicts. So I think ability to choose the variables 
> names and provide them is overkill.

This is maybe a bit contrived, but one benefit of some configurability
is that if you have a format with two of the same checksums for
different parts of the data, you don't need newVariableInstance stuff.
For example:

  
  

  

  


  

  

So it's just a bit cleaner looking. Though, I'm not sure that's a strong
argument for configuring the variables. I imagine in most formats where
there's multiple of the same checksums then it's in an array and you'd
need new variable instance since the number of checksums isn't known.


> I think of the variable definitions as coming from an imported schema that 
> one must have to use the layer transform.
> Right now we don't have a way of declaring a layer transform when defined 
> outside of the daffodil code base in a pluggable fashion, but assume we had 
> something like  className="com.myco.foobar"/> which would
> also appear in that import file, then accessing and using the layer transform 
> and its associated variables would all be obtained from the one import 
> statement.
> 
> 2) For the IPv4 layer, it feels a bit unfortunate to have to split the
> CRC into two separate layers, since the CRC algorithm is really just a
> checksum over the whole header with just the checksum field treated as
> if it were zero. Is it possible to have a property that just specifies
> that the Nth byte doesn't contribute? Maybe something like:
> 
>dfdlx:runtimeProperties="ignoreByte=5">...
> 
> @@@ In the case of the IPv4 checksum, it can just hardcode the fact that it 
> skips those specific bytes.  I included the splitting into two separate 
> layers just to illustrate that this complexity could be handled. I will look 
> at recasting this as just one checksum layer and see how it comes out. I 
> think the other example of the GPS data format with parity bit computations, 
> is worth looking at as that one is fairly complicated in which bits 
> contribute in what ways.

Agreed. I wasn't sure if there the IPv4 checks is specific to IPv4 or if
there are other uses where different bytes (or no bytes) are ignored.
Just thinking about re usability, but that's maybe more of an
implementation detail.

I'll take a look at the GPS example.

> 3) As for implementing the checksums, have you put any thought into
> making that extensible? For example, I'm wondering if we only have a
> single "checksum" layer, and then the dfdlx:runtimeProperties determines
> which algorithm to use? E.g.
> 
>dfdlx:runtimeProperties="algorithm=crc32">...
> 
>dfdlx:runtimeProperties="algorithm=ipv4header">...
> 
> And then people can register different checksum algorithms without
> having to reimplement their own layer? Or maybe we keep it simple and
> the default checksum layer just supports a handful of the most common
> checksums (maybe those supported by some preexisting checksum library?)
> 
> People could still implement their own pluggable checksum layer if they
> need something we don't support, but this would cover the most common
> cases and avoids a proliferation of a bunch of different layers that are
> basically the same except for some minor algorithm details.
> 
> @@@ This refactoring can of course be done. But isn't needed to get started. 
> Parameters to transform algorithms can be passed in variables, or could be 
> specified using an extensible property bag such as dfdlx:runtimeProperties as 
> you have shown. We may want a dedicated dfdl:layerParameters property since 
> we have other layering-specific properties (e.g., for layering length kind, 
> etc.) rather than using a generic hook. Ideally layering transformers could 
> check these properties statically and issue SDEs if misused.

Agreed.

> 
> On 7/30/21 2:29 PM, Beckerle, Mike wrote:
>> I would like comments on the layering enhancement to enable checksum
>> computations in DFDL schemas.
>>
>>
>> This is a high-priority feature for Daffodil's next release 3.2.0, especially
>> for cybersecurity applications of Daffodil, which I know a number of us are
>> 

Re: Please review mock up idea for checksum calculations in DFDL

[Beckerle, Mike]

Comments below see @@@mb


From: Steve Lawrence 
Sent: Monday, August 9, 2021 12:18 PM
To: dev@daffodil.apache.org 
Subject: Re: Please review mock up idea for checksum calculations in DFDL

Some comments:

1) I like the idea that the layers write to a variable, but it seems
like the variables are hard coded in the layer transformer? What are
your thoughts on having the variable defined in a property so that the
user has more control over the naming/definition of it, maybe via
something like dfdlx:runtimeProperties? For example:

  ...

@@@ given that a layer transform can be defined with a unique namespace defined 
by way of a URI, there's never a need to be
concerned about naming conflicts. So I think ability to choose the variables 
names and provide them is overkill.

I think of the variable definitions as coming from an imported schema that one 
must have to use the layer transform.
Right now we don't have a way of declaring a layer transform when defined 
outside of the daffodil code base in a pluggable fashion, but assume we had 
something like  
which would
also appear in that import file, then accessing and using the layer transform 
and its associated variables would all be obtained from the one import 
statement.

2) For the IPv4 layer, it feels a bit unfortunate to have to split the
CRC into two separate layers, since the CRC algorithm is really just a
checksum over the whole header with just the checksum field treated as
if it were zero. Is it possible to have a property that just specifies
that the Nth byte doesn't contribute? Maybe something like:

  ...

@@@ In the case of the IPv4 checksum, it can just hardcode the fact that it 
skips those specific bytes.  I included the splitting into two separate layers 
just to illustrate that this complexity could be handled. I will look at 
recasting this as just one checksum layer and see how it comes out. I think the 
other example of the GPS data format with parity bit computations, is worth 
looking at as that one is fairly complicated in which bits contribute in what 
ways.

3) As for implementing the checksums, have you put any thought into
making that extensible? For example, I'm wondering if we only have a
single "checksum" layer, and then the dfdlx:runtimeProperties determines
which algorithm to use? E.g.

  ...

  ...

And then people can register different checksum algorithms without
having to reimplement their own layer? Or maybe we keep it simple and
the default checksum layer just supports a handful of the most common
checksums (maybe those supported by some preexisting checksum library?)

People could still implement their own pluggable checksum layer if they
need something we don't support, but this would cover the most common
cases and avoids a proliferation of a bunch of different layers that are
basically the same except for some minor algorithm details.

@@@ This refactoring can of course be done. But isn't needed to get started. 
Parameters to transform algorithms can be passed in variables, or could be 
specified using an extensible property bag such as dfdlx:runtimeProperties as 
you have shown. We may want a dedicated dfdl:layerParameters property since we 
have other layering-specific properties (e.g., for layering length kind, etc.) 
rather than using a generic hook. Ideally layering transformers could check 
these properties statically and issue SDEs if misused.


On 7/30/21 2:29 PM, Beckerle, Mike wrote:
> I would like comments on the layering enhancement to enable checksum
> computations in DFDL schemas.
>
>
> This is a high-priority feature for Daffodil's next release 3.2.0, especially
> for cybersecurity applications of Daffodil, which I know a number of us are
> involved in.
>
>
> I've produced a mock-up of how it would look, with lots of annotations in a 
> WIP
> pull request on the ethernetIP DFDL schema. I only did the mock-up for the 
> IPV4
> element, so look at that element in the ethernetIP.dfdl.xsd.
>
> (UDP and TCP packets have their own additional checksums - I didn't mock up
> those, just IPV4)
>
>
> This is at https://github.com/DFDLSchemas/ethernetIP/pull/1
> 
>
>
> This doesn't run, it's just an initial mock-up of the ideas for
> checksum/CRC/parity recomputation capability as a further simple extension of
> the existing DFDL layering extension.
>
>
> The layering extension itself is described here:
>
> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Data+Layering+for+Base64%2C+Line-Folding%2C+Compression%2C+Etc
> 
>
>
> I did notice that none of the published DFDLSchemas actually use the layering
> transforms that we've built into Daffodil. There are some non-public DFDL
> schemas that do use this extension to do line-folding transformations.
>
>
> There are, however, tests