Mike, I'll take a look. When you say these pull requests aren't working yet, what are the missing parts? I know at a minimum you need to implement the following transform names:
<dfdl:format layerTransform="checksumLayerPart1" layerLengthKind="implicit"/> <dfdl:format layerTransform="checksumLayerPart2" layerLengthKind="implicit"/> <dfdl:format layerTransform="gpssps:parityLayer" layerLengthKind="implicit"/> <dfdl:format layerTransform="gpssps:parityLayerHOW" layerLengthKind="implicit"/> Do you plan to implement these transform names in Scala within the Daffodil codebase or as pluggable algorithms akin to UDFs? Do you need to implement or change anything else in the layering functionality, given that the following initial transform names already are enumerated in daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/DFDL_part1_simpletypes.xsd? * fourbyteswap: Swap bytes in 32 bit words * base64_MIME: IETF RFC 2045, max line length 76 characters * base64: IETF RFC 4648 - not the URL-SAFE version * base64url: IETF RFC 4648 - the URL-SAFE version * lineFolded_IMF: IETF RFC 2822 Internet Message Format (IMF) * lineFolded_iCalendar: IETF RFC 5545 Internet Calendaring and Scheduling (iCalendar) * quotedPrintable: IETF RFC 2045 Quoted Printable Content Transfer Encoding * aisPayloadArmor: Automatic Identification System - ITU-R M.1371-1 * compress: Lempel-Ziv-Welch compression per Unix 'compress' command * gzip: GZIP per https://www.ietf.org/rfc/rfc1952.txt Actually, I don't see implementations of all these transform names - base64, base64url, and compress still seem to be unimplemented in the Daffodil codebase at this time. But anyway, is all you need to do is to add these new transform names to the enumerated transform names and write the corresponding LayerTransformer & LayerTransformerFactory classes, tests, etc.? I also saw Steve's comments and I think he has some good points on stuff like combining checksumLayerPart1 and checksumLayerPart2 into a single checksumLayer with a parameter specifying where the checksum field is so its value can be zero during the checksum computation. John From: Beckerle, Mike <mbecke...@owlcyberdefense.com> Sent: Tuesday, August 3, 2021 4:36 PM To: dev@daffodil.apache.org Subject: EXT: Please review: DFDL parity calculations also - was: Fw: Please review mock up idea for checksum calculations in DFDL A second example focused on DFDL with parity calculations in a GPS format has also been "mocked up" https://github.com/DFDLSchemas/gps-sps/pull/1 Please review and comment on this pull request also. The GPS spec this is based on is in the repository also in the doc directory. Thank you ________________________________ From: Beckerle, Mike Sent: Friday, July 30, 2021 2:29 PM To: dev@daffodil.apache.org<mailto:dev@daffodil.apache.org> <dev@daffodil.apache.org<mailto:dev@daffodil.apache.org>> Subject: Please review mock up idea for checksum calculations in DFDL I would like comments on the layering enhancement to enable checksum computations in DFDL schemas. This is a high-priority feature for Daffodil's next release 3.2.0, especially for cybersecurity applications of Daffodil, which I know a number of us are involved in. I've produced a mock-up of how it would look, with lots of annotations in a WIP pull request on the ethernetIP DFDL schema. I only did the mock-up for the IPV4 element, so look at that element in the ethernetIP.dfdl.xsd. (UDP and TCP packets have their own additional checksums - I didn't mock up those, just IPV4) This is at https://github.com/DFDLSchemas/ethernetIP/pull/1 This doesn't run, it's just an initial mock-up of the ideas for checksum/CRC/parity recomputation capability as a further simple extension of the existing DFDL layering extension. The layering extension itself is described here: https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Data+Layering+for+Base64%2C+Line-Folding%2C+Compression%2C+Etc I did notice that none of the published DFDLSchemas actually use the layering transforms that we've built into Daffodil. There are some non-public DFDL schemas that do use this extension to do line-folding transformations. There are, however, tests showing the DFDL layering extension in daffodil's code base. See https://github.com/apache/daffodil/blob/master/daffodil-test/src/test/resources/org/apache/daffodil/layers/layers.tdml and search for dfdlx:layerTransform property. The mock-up effectively proposes allowing layer transforms to read and write DFDL variables, as a means of them accepting input parameters, and as the means of them computing and returning output results. I plan to do a couple other mock-ups of a check-digit calculation, and some parity bit computations, but this IPV4 is enough to get the gist of the idea. I'd appreciate feedback on this, which you can do on the pull request in the usual github code review manner. -mikeb Mike Beckerle | Principal Engineer [cid:2b10f593-ca11-4030-8f7b-3db1a1024055] mbecke...@owlcyberdefense.com<mailto:bhum...@owlcyberdefense.com> P +1-781-330-0412