If we look at the schema for the TCP header:
<xs:complexType>
<xs:sequence>
<xs:element name="PortSRC" type="pcap:bit" dfdl:length="16" />
<xs:element name="PortDest" type="pcap:bit" dfdl:length="16" />
<xs:element name="Seq" type="pcap:bit" dfdl:length="32" />
<xs:element name="Ack" type="pcap:bit" dfdl:length="32" />
<xs:element name="DataOffset" type="pcap:bit" dfdl:length="4"
dfdl:outputValueCalc="{ (dfdl:valueLength(../Options, 'bytes')
div 4) + 5 }" />
<xs:element name="Reserved" type="pcap:bit" dfdl:length="3" />
<xs:element name="Flags" type="pcap:bit" dfdl:length="9" />
<xs:element name="WindowSize" type="pcap:bit" dfdl:length="16" />
<xs:element name="Checksum" type="pcap:bit" dfdl:length="16" />
<xs:element name="Urgent" type="pcap:bit" dfdl:length="16" />
<xs:element name="Options" type="pcap:hexByte" dfdl:length="{
(xs:unsignedInt(../DataOffset) - 5) * 4 }" />
</xs:sequence>
</xs:complexType>
So I was incorrect about byte orientation.
Note that element DataOffset is 4 bits wide, and has outputValueCalc which
wants valueLength of ../Options in bytes, then does div 4 + 5.
The next element Reserved is 3 bits wide
The next element Flags is 9 bits.
In theory, if the TCP header started on a byte boundary, it would now be back
on a byte boundary, as 4 + 3 +9 is 16 bits.
The next element WindowSize is 16 bits
The next element Checksum, is 16 bits
The next element Urgent is 16 bits
Then we have Options, which is hexBinary, the target length (for unparsing)
given by ( ../DataOffset - 5) * 4.
This is about as tricky as it gets in unparsing. We have a non-circular
definition here because the DataOffset is computed in terms of the
dfdl:valueLength of ../Options, where the target length for ../Options is
computed in terms of ../DataOffset. The target length is needed compute the
content length, but not to compute the value length.
I will start by creating a test that isolates this TCP Header.
________________________________
From: Mike Beckerle
Sent: Thursday, December 21, 2017 9:28:07 AM
To: Steve Lawrence; [email protected]
Subject: Re: latest pushes to PR 12
Ok, so the challenge is in getting this error reproducible in a smaller example.
Since these are bytes in the middle of hexBinary data, or in the middle of a
16-bit bigEndian integer, that limits the possibilities for where the error
could be introduced.
Also, If i recall correctly, PCAP is all byte-oriented. There are no bit-fields
that are not a mulitple of 8 bits wide. There are no 4-bit wide fields even. I
will look at this more carefully, because without this I am really at a loss as
to what could be it.
However, PCAP does use outputValueCalc extensively. So when unparsing there are
split data streams that are being reassembled. I just don't know how this could
result in something like this happening in the middle of hexBinary data..., or
in the middle (2nd byte) of a 16 bit integer.
....mikeb
________________________________
From: Steve Lawrence <[email protected]>
Sent: Thursday, December 21, 2017 8:32:38 AM
To: [email protected]; Mike Beckerle
Subject: Re: latest pushes to PR 12
I took at a look at the pcap file that is failing and I've tracked down
the first two differences in the file. I'm not sure where the problem is
in Daffodil, but this should help to narrow down the search.
The first packet is unparsed correctly. The second and third packets are
not. The Options field in the TCPHeader of the second packet is:
<Options>0204059804020000</Options>
This is exactly what the data looks like in the original file starting
at byte position 224 (0xEO in hex). When this blob is unparsed, it
results in the following hex:
02 04 05 F8 04 02 00 00
Note the only difference here is the 4th byte which is 0xF8 instead of
0x98. So the upper nibble of that byte is 0b1111 instead of 0b1001. The
type of Options is xs:hexBinary.
The second incorrect byte is in the Checksum field of the third packet
(byte position 318, 0x13E in hex). The correct value is (16-bit BE
xs:unsignedInt) 22312 but is unparsed as 22520. Converted to hex these
values are:
22312: 0x5728
22520: 0x57F8
So again, we see some upper nibble converted to 0b1111. It's not clear why.
Running a binary diff on the correct output and the unparsed output
shows the the differences are always some upper nibble (the value
varies) being converted to 0b1111. So something in the unparse logic
isn't handling this upper nibble correctly. In one case it was an
xs:hexBinary blob, in the other case it was a 2-byte integer. So maybe
the bug is in is some shared logic in unparsing hexBinary and integers,
or perhaps maybe related to collapsing buffered DOS's to direct? It's
not exactly clear, and it doesn't happen all that often (I think there's
only 8 occurrences in this file). But when it does happen, it's always
some upper nibble converted to 0b1111.
I'll continue investigating, but hopefully this should narrow down where
we need to be looking.
- Steve
This is just a hex binary blob of data.
On 12/20/2017 07:21 PM, Mike Beckerle wrote:
> So I have fixed all the issues with PR 12 save 1.
>
>
> I added 4 commits. Granular is better with this review system I've determined.
>
>
> Any insight into what changed that caused the need for the changes to
> InteractiveDebugger.scala in the last (6th) commit... are welcome. I didn't
> like putting this hack in.
>
>
> Note: this branch is yet to be rebased onto master. I still have that todo.
>
>
> Status is all tests pass, sbt cli all tests pass, all DFDLSchemas and fouo
> schemas pass, except 1:
>
>
> PCAP test_pcap_test_http_ipv6 still fails.
>
>
> Steve L - can we use wireshark or some other tool to verify that the
> checksums in the expected infoset XML for this test are in fact correct?
>
> I don't want to debug in detail if these were never verified before.
>
>
> I still have these todos:
>
> * changes to make per review comments on the 2nd commit
> * rebase onto master.
> * PCAP test failure mystery
>
>
