RE: Please review: DFDL parity calculations also - was: Fw: Please review mock up idea for checksum calculations in DFDL

2021-08-10 Thread Interrante, John A (GE Research, US) 
Mike,

I'll take a look.  When you say these pull requests aren't working yet, what 
are the missing parts?  I know at a minimum you need to implement the following 
transform names:







Do you plan to implement these transform names in Scala within the Daffodil 
codebase or as pluggable algorithms akin to UDFs?   Do you need to implement or 
change anything else in the layering functionality, given that the following 
initial transform names already are enumerated in 
daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part1_simpletypes.xsd?


  *   fourbyteswap: Swap bytes in 32 bit words
  *   base64_MIME: IETF RFC 2045, max line length 76 characters
  *   base64: IETF RFC 4648 - not the URL-SAFE version
  *   base64url: IETF RFC 4648 - the URL-SAFE version
  *   lineFolded_IMF: IETF RFC 2822 Internet Message Format (IMF)
  *   lineFolded_iCalendar: IETF RFC 5545 Internet Calendaring and Scheduling 
(iCalendar)
  *   quotedPrintable: IETF RFC 2045 Quoted Printable Content Transfer Encoding
  *   aisPayloadArmor: Automatic Identification System - ITU-R M.1371-1
  *   compress: Lempel-Ziv-Welch compression per Unix 'compress' command
  *   gzip: GZIP per https://www.ietf.org/rfc/rfc1952.txt

Actually, I don't see implementations of all these transform names - base64, 
base64url, and compress still seem to be unimplemented in the Daffodil codebase 
at this time.  But anyway, is all you need to do is to add these new transform 
names to the enumerated transform names and write the corresponding 
LayerTransformer & LayerTransformerFactory classes, tests, etc.?

I also saw Steve's comments and I think he has some good points on stuff like 
combining checksumLayerPart1 and checksumLayerPart2 into a single checksumLayer 
with a parameter specifying where the checksum field is so its value can be 
zero during the checksum computation.

John

From: Beckerle, Mike 
Sent: Tuesday, August 3, 2021 4:36 PM
To: dev@daffodil.apache.org
Subject: EXT: Please review: DFDL parity calculations also - was: Fw: Please 
review mock up idea for checksum calculations in DFDL

A second example focused on DFDL with parity calculations in a GPS format has 
also been "mocked up"

https://github.com/DFDLSchemas/gps-sps/pull/1

Please review and comment on this pull request also. The GPS spec this is based 
on is in the repository also in the doc directory.

Thank you


From: Beckerle, Mike
Sent: Friday, July 30, 2021 2:29 PM
To: dev@daffodil.apache.org<mailto:dev@daffodil.apache.org> 
mailto:dev@daffodil.apache.org>>
Subject: Please review mock up idea for checksum calculations in DFDL


I would like comments on the layering enhancement to enable checksum 
computations in DFDL schemas.



This is a high-priority feature for Daffodil's next release 3.2.0, especially 
for cybersecurity applications of Daffodil, which I know a number of us are 
involved in.



I've produced a mock-up of how it would look, with lots of annotations in a WIP 
pull request on the ethernetIP DFDL schema. I only did the mock-up for the IPV4 
element, so look at that element in the ethernetIP.dfdl.xsd.

(UDP and TCP packets have their own additional checksums - I didn't mock up 
those, just IPV4)



This is at https://github.com/DFDLSchemas/ethernetIP/pull/1



This doesn't run, it's just an initial mock-up of the ideas for 
checksum/CRC/parity recomputation capability as a further simple extension of 
the existing DFDL layering extension.



The layering extension itself is described here:

https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Data+Layering+for+Base64%2C+Line-Folding%2C+Compression%2C+Etc



I did notice that none of the published DFDLSchemas actually use the layering 
transforms that we've built into Daffodil. There are some non-public DFDL 
schemas that do use this extension to do line-folding transformations.



There are, however, tests showing the DFDL layering extension in daffodil's 
code base. See
https://github.com/apache/daffodil/blob/master/daffodil-test/src/test/resources/org/apache/daffodil/layers/layers.tdml
and search for dfdlx:layerTransform property.



The mock-up effectively proposes allowing layer transforms to read and write 
DFDL variables, as a means of them accepting input parameters, and as the means 
of them computing and returning output results.



I plan to do a couple other mock-ups of a check-digit calculation, and some 
parity bit computations, but this IPV4 is enough to get the gist of the idea.



I'd appreciate feedback on this, which you can do on the pull request in the 
usual github code review manner.



-mikeb



Mike Beckerle | Principal Engineer

[cid:2b10f593-ca11-4030-8f7b-3db1a1024055]

mbecke...@owlcyberdefense.com<mailto:bhum...@owlcyberdefense.com>
P +1-781-330-0412

Please review: DFDL parity calculations also - was: Fw: Please review mock up idea for checksum calculations in DFDL

2021-08-03 Thread Beckerle, Mike 
A second example focused on DFDL with parity calculations in a GPS format has 
also been "mocked up"

https://github.com/DFDLSchemas/gps-sps/pull/1

Please review and comment on this pull request also. The GPS spec this is based 
on is in the repository also in the doc directory.

Thank you


From: Beckerle, Mike
Sent: Friday, July 30, 2021 2:29 PM
To: dev@daffodil.apache.org 
Subject: Please review mock up idea for checksum calculations in DFDL


I would like comments on the layering enhancement to enable checksum 
computations in DFDL schemas.


This is a high-priority feature for Daffodil's next release 3.2.0, especially 
for cybersecurity applications of Daffodil, which I know a number of us are 
involved in.


I've produced a mock-up of how it would look, with lots of annotations in a WIP 
pull request on the ethernetIP DFDL schema. I only did the mock-up for the IPV4 
element, so look at that element in the ethernetIP.dfdl.xsd.

(UDP and TCP packets have their own additional checksums - I didn't mock up 
those, just IPV4)


This is at https://github.com/DFDLSchemas/ethernetIP/pull/1


This doesn't run, it's just an initial mock-up of the ideas for 
checksum/CRC/parity recomputation capability as a further simple extension of 
the existing DFDL layering extension.


The layering extension itself is described here:

https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Data+Layering+for+Base64%2C+Line-Folding%2C+Compression%2C+Etc


I did notice that none of the published DFDLSchemas actually use the layering 
transforms that we've built into Daffodil. There are some non-public DFDL 
schemas that do use this extension to do line-folding transformations.


There are, however, tests showing the DFDL layering extension in daffodil's 
code base. See

https://github.com/apache/daffodil/blob/master/daffodil-test/src/test/resources/org/apache/daffodil/layers/layers.tdml
and search for dfdlx:layerTransform property.


The mock-up effectively proposes allowing layer transforms to read and write 
DFDL variables, as a means of them accepting input parameters, and as the means 
of them computing and returning output results.


I plan to do a couple other mock-ups of a check-digit calculation, and some 
parity bit computations, but this IPV4 is enough to get the gist of the idea.


I'd appreciate feedback on this, which you can do on the pull request in the 
usual github code review manner.


-mikeb



Mike Beckerle | Principal Engineer

[cid:2b10f593-ca11-4030-8f7b-3db1a1024055]

mbecke...@owlcyberdefense.com

P +1-781-330-0412