Please review mock up idea for checksum calculations in DFDL

[Beckerle, Mike]

I would like comments on the layering enhancement to enable checksum 
computations in DFDL schemas.


This is a high-priority feature for Daffodil's next release 3.2.0, especially 
for cybersecurity applications of Daffodil, which I know a number of us are 
involved in.


I've produced a mock-up of how it would look, with lots of annotations in a WIP 
pull request on the ethernetIP DFDL schema. I only did the mock-up for the IPV4 
element, so look at that element in the ethernetIP.dfdl.xsd.

(UDP and TCP packets have their own additional checksums - I didn't mock up 
those, just IPV4)


This is at https://github.com/DFDLSchemas/ethernetIP/pull/1


This doesn't run, it's just an initial mock-up of the ideas for 
checksum/CRC/parity recomputation capability as a further simple extension of 
the existing DFDL layering extension.


The layering extension itself is described here:

https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Data+Layering+for+Base64%2C+Line-Folding%2C+Compression%2C+Etc


I did notice that none of the published DFDLSchemas actually use the layering 
transforms that we've built into Daffodil. There are some non-public DFDL 
schemas that do use this extension to do line-folding transformations.


There are, however, tests showing the DFDL layering extension in daffodil's 
code base. See

https://github.com/apache/daffodil/blob/master/daffodil-test/src/test/resources/org/apache/daffodil/layers/layers.tdml
and search for dfdlx:layerTransform property.


The mock-up effectively proposes allowing layer transforms to read and write 
DFDL variables, as a means of them accepting input parameters, and as the means 
of them computing and returning output results.


I plan to do a couple other mock-ups of a check-digit calculation, and some 
parity bit computations, but this IPV4 is enough to get the gist of the idea.


I'd appreciate feedback on this, which you can do on the pull request in the 
usual github code review manner.


-mikeb



Mike Beckerle | Principal Engineer

[cid:2b10f593-ca11-4030-8f7b-3db1a1024055]

mbecke...@owlcyberdefense.com

P +1-781-330-0412

Re: trying to rerun checks on PR - no option for it?

[Beckerle, Mike]

Ok, so to me this restriction is just plain wrong.

A project that has multiple branches moving forward can't even work this way at 
all.

The tests to run are well defined for this change set.

What if we decided to never merge it back to master, but instead start carrying 
forward a dev branch separate from master?

I don't want any contributor to have to rebase onto latest master until we see 
that their stuff works, passes the checks, etc. Rebasing on top of master will 
lose the history of comments on the PR. (all the commits change). That's also a 
"bug" not a feature, but to live with that we have to simply not rebase until 
very late in the game, and the current policy makes the automated CI testing 
only workable if you rebase onto master frequently, which we know is flawed.

Am I incorrect here?






From: Interrante, John A (GE Research, US) 
Sent: Friday, July 30, 2021 10:59 AM
To: dev@daffodil.apache.org 
Subject: RE: trying to rerun checks on PR - no option for it?

CI won't run when the PR has conflicts with the main branch.  That's why the 
button isn't there.  Darryl needs to rebase his PR, fix the conflict, and push 
his PR again with " git push --force-with-lease".

From: Beckerle, Mike 
Sent: Friday, July 30, 2021 10:51 AM
To: dev@daffodil.apache.org
Subject: EXT: trying to rerun checks on PR - no option for it?

Darryl S. pushed a commit to his PR 
https://github.com/apache/daffodil/pull/601/checks

As a first time contributor, his checks won't automatically run.

I was going to trigger them manually, but I see no option for doing so.

Wasn't there a button for that? In the past I swear I saw one.

Anybody understand what's up with this before I open an INFRA ticket?


Mike Beckerle | Principal Engineer

[cid:9a6b4607-36da-4f6f-9218-104142fd8991]

mbecke...@owlcyberdefense.com
P +1-781-330-0412

RE: trying to rerun checks on PR - no option for it?

[Interrante, John A (GE Research, US)]

CI won't run when the PR has conflicts with the main branch.  That's why the 
button isn't there.  Darryl needs to rebase his PR, fix the conflict, and push 
his PR again with " git push --force-with-lease".

From: Beckerle, Mike 
Sent: Friday, July 30, 2021 10:51 AM
To: dev@daffodil.apache.org
Subject: EXT: trying to rerun checks on PR - no option for it?

Darryl S. pushed a commit to his PR 
https://github.com/apache/daffodil/pull/601/checks

As a first time contributor, his checks won't automatically run.

I was going to trigger them manually, but I see no option for doing so.

Wasn't there a button for that? In the past I swear I saw one.

Anybody understand what's up with this before I open an INFRA ticket?


Mike Beckerle | Principal Engineer

[cid:9a6b4607-36da-4f6f-9218-104142fd8991]

mbecke...@owlcyberdefense.com
P +1-781-330-0412

trying to rerun checks on PR - no option for it?

[Beckerle, Mike]

Darryl S. pushed a commit to his PR 
https://github.com/apache/daffodil/pull/601/checks

As a first time contributor, his checks won't automatically run.

I was going to trigger them manually, but I see no option for doing so.

Wasn't there a button for that? In the past I swear I saw one.

Anybody understand what's up with this before I open an INFRA ticket?


Mike Beckerle | Principal Engineer

[cid:9a6b4607-36da-4f6f-9218-104142fd8991]

mbecke...@owlcyberdefense.com

P +1-781-330-0412