On 24 July 2018 at 11:46, Guy Harris <ghar...@sonic.net> wrote: > On Jul 23, 2018, at 6:36 PM, David Mirabito <dav...@metamako.com> wrote: > > > We'd like to request a new DLT_ value for Metamako's packet capture > trailer > > that is generated by our MetaWatch network application. > > > > https://www.metamako.com/applications/metawatch-app.html > > > > It is a variable length data trailer, appended immediately after an > > original Ethernet Payload. It is backwards-compatible with DLT_EN10MB but > > adds additional fields after the original frame payload until the > > end-of-frame marker. > > > > We were thinking something like this > > > > LINKTYPE_METAMAKO_TRAILER --> DLT_METAMAKO_TRAILER > > > > Is there anything in particular that you require? > > A detailed description of the format of the trailer. > > Presumably, for Ethernet packets with a type field, the trailer can't be > found until the payload is dissected, because there's no other way to > determine how large the payload is. > > Also, presumably, the trailer comes after any padding, and there's no FCS > in these packets.
Hi Guy, I have attached a more detailed description as a text file (lest email mangle the ASCII-art), but in short, a packet would look on the wire like: * SOF * Destination address * Source Address * EtherType/Length * Payload * FCS . <--- everything up to, and including, here is unmodified by MetaWatch. Typically our appending a trailer may allow this FCS to be captured by NICs which would otherwise have stripped it * Optional Extension TLVs * Timestamp: Seconds * Timestamp: Nanoseconds * Flags (currently whether FCS was correct at ingress, and presence of any extension TLVs) * Device ID (by default unique portion of the device's serial number) * Port ID * NewFCS (calculated so that metawatch emits a valid frame, not always captured by capturing NICs) * EOF The intent when parsing is that the base trailer is always the last 12 bytes so given a complete capture one may simply skip directly to "wlen - 12", so there is no need to interpret EtherLen, or do a full dissection to locate the trailer. I agree that the case where caplen < wirelen may be problematic, but then you're guaranteed a corrupted or missing trailer anyway so it might not be worth even trying. Our current Wireshark integration does indeed register as a heuristic Ethernet Trailer, and as such does rely on a correct prior dissection to identify any trailer and give us a chance to match. We tend ot make use of the "Fixed ethernet trailer length" feature in the cases where prior dissection fails to identify any trailer. The main motivation to request a DLT that this allows us first-chance look at each frame in capture known to contain trailers, in part to eliminate such reliance. We then may scrape the trailer off the end, and then pass the remainder down the usual dissector stack. To me it seemed similar to the pseudo-header notion, except working backwards from the buffer's end. It also allows us to poke our timestamp in as "the" time used for display and any measurements (i.e RTT) done by sub-dissectors when opening such a trace file. Regarding padding: * In a compliant frame, the original sender would have padded and then added an FCS (covering the padding) on the wire. To us this looks like any other 64-byte frame and we still immediately stick the trailer immediately after this original FCS * In the case we are presented with an undersized frame, we will add enough padding ourselves such that "whatever came in + padding + tailer" will be a legal frame. Presumably in this case the original FCS would not have been correct, but the one we apply at the end will be correct, regardless. Either way this would preserve the parsing scheme. The "original FCS" may be flagged as incorrect in the case of runts or other bit errors prior to our ingress, but we still "wrap" in a trailer + good new-FCS allowing such frames to be captured on commodity HW. I hope this helps, please let us know any more info might be required. Thanks, David M
Metamako Trailer. ================ This describes a potentially variable length data trailer, appended immediately after an original Ethernet Payload. Being a trailer, it is backwards-compatible with DLT_EN10MB: an unaware decoder will simply see extra bytes between the end of the original frame payload and the end-of-frame marker. These should be safely ignored as in the case of padded frames anyway. The trailer is intended to be walked backwards from the End of Frame (EOF). Unless otherwise specified, all multi-byte values are in network-byte-order (Big Endian). Base Trailer ------------ At a minimum, all Ethernet frames are grown by 16 bytes with the addition of the trailer structure immediately after the FCS of the timestamped frame. Due to the varying length of the Ethernet payload, the end of the OriginalFCS (and hence the start of any subsequent trailer data) may no be 32bit aligned. Trailer data always immediately follows the FCS and not padding is inserted. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------------------------------------------------------+ | Seconds | +---------------------------------------------------------------+ | Nanoseconds | +---------------------------------------------------------------+ | Reserved |X|V| DeviceID | PortID | +---------------------------------------------------------------+ | NewFCS | +---------------------------------------------------------------+ Seconds: Time in seconds (UTC) since the Jan 1st, 1970 Epoch. Nanoseconds: Nanosecond portion of time. Will be less than 1,000,000,000 Reserved: Reserved for future use, to be written as zero and ignored if set eXtensions present 1: If optional trailer extensions are present. 0: If no extensions are present. FCS Valid: 1: If the original FCS of the frame was correct. 0: If the original FCS was incorrect. DeviceID: A 16bit identifier for the device which captured this frame. Defaults to the unique portion of the device's serial number but may be configured to any value. PortID: An 8bit identifier for the port which captured this frame. Defaults to the number of the ingress AP port, but may be configured to any value. NewFCS: A new FCS is calculated and appended so that the annotated Ethernet frame + original FCS + [extensions] + trailer is still valid on the wire. NB: Not all capturing devices will record this field, presumably those would also not have recorded the FCS value of the un-trailered Ethernet frame either. Trailer Extensions ================== When present, trailers are found immediately preceding the 'Seconds' field of the base trailer, in TLV (Tag, Length, Value) format. Currently only some tag types are defined, decoders should use the length field to ignore and skip over any unknown extension types. There are two tag formats, 'primary' and 'secondary' which offer a tradeoff between compactness and the ability to select from more available tags or longer payload lengths. Primary ------- Primary tags are the most compact, allowing up to 24bits of payload to be specified in the Tag header. Up to 3 additional 32bit words of payload may also be specified. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------------------------------------------------------+ | Data |Len|F| Tag | +---------------------------------------------------------------+ Tag: Specified the type of information stored in the Data section. Allowable values are 0x00-0x1E (0-30). Tag value 0x1F (31) is used to note a Secondary extension type Final: 1: If this is the Final extension present. 0: If there is another extension preceding this one's data. Trailers and extensions are processed walking backwards from the EOF, so the first extension received on the wire will be last one processed and hence have this bit set. Only one extension header may have the Final bit set. Length: The number of additional 32bit words of payload in this extension. Note that a value of zero is allowed, as the inline 24bits may be sufficient and no additional words may be required. Data: Either 24, 56, 88 or 120 bits of Tag-specific data, depending on the Length value. Secondary --------- Secondary tags allow more possible Type values and longer possible lengths buy not having any data payload inline in the extension header. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------------------------------------------------------+ | Data | +---------------------------------------------------------------+ | Tag2 | Len |F| 11111 | +---------------------------------------------------------------+ Tag: Always 0x1F (32) in secondary tags Final: Same meaning as Primary extensions Len: Value is N-1, specifying how many additional 32bit words of Data payload preceed the extension header 0 --> 1 word (32bits) 1 --> 2 words (64 bits) ... 1023 --> 1024 words (32768 bits) Tag2: Specifies the meaning of the Data payload. Defined extension tags ======================= Some extension types have already been defined: Sequence number: ---------------- * Primary Extension; Tag = 1 Sequence Numbers are unsigned 16bit numbers, maintained per-port and incremented on ingress. Gaps in a port's numbers may indicate drops. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------------------------------------------------------+ | Reserved | SequenceNo |Len|F| 00000 | +---------------------------------------------------------------+ Fractional Nanoseconds: ----------------------- * Primary Extension; Tag = 2 Additional timestamp precision may be available beyond the nanoseconds-level found in the base trailer. Value is in 1/2^24 of 1ns. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------------------------------------------------------+ | FractionalNS |Len|F| 00001 | +---------------------------------------------------------------+ ASCII String: ------------ * Secondary Extension; Tag2 = 00 Allows for arbitrary strings up to 4k in length. Strings need not be zero-terminated if they end on a 4-byte boundary, and unused bytes should be padded with zero. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------------------------------------------------------+ | | + ASCII String + | | +---------------------------------------------------------------+ | 0000000000000000 | Len |F| 11111 | +---------------------------------------------------------------+ Example ======= Here we show a full frame on the wire: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DestinationAddress | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | SourceAddress | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | EtherType | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + | | + Payload + ... + (46-1500bytes) + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | OrigFCS | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 0x064dd3 | 00|1| 00001 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 0x00 | 0xffb6 | 00|0| 00000 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 0x5b32cdda | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 0x2a2b1a86 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 000000 |1|1| 0x1dc0 | 0x09 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | NewFCS | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The decoding process is as follows: 1. The NewFCS may or may not by physically captured. Regardless, we can assume is correct by virtue of successfully capturing the frame. 2. Flags, DeviceID and PortID fields are always in a fixed location relative to the end of frame. We determine: * The original FCS was good. (i.e running CRC32 over from DestinationAddr to the end of Ethernet payload would yield OrigFCS) * There is at least one Trailer Extension present. * Packet was timestamped by device 7617 on Application Port #9. 3. Similarly, Seconds:Nanoseconds fields are also fixed relative to EOF: * The timestamp is 1530056154.707467910 UTC --> Jun 27, 2018 09:35:54.707467910 AEST 4. An extension is present so the 32bits prior to Seconds field is the header of the first extension. * Type is 0 --> is sequence number #65462 * Final is not set --> there is at least one more trailer extension present * Length is 0, no more words for this tag. SequenceNumbers are defined to have a length of 0, but unaware implementations may use the Length field to skip to the next extension header. 5. The next extension header in in the preceding 32bit word * Type is 1 --> Is a Fractional Nanosecond additional timestamp precision * Final is set --> no more extensions to process. * Length is 0 --> no more data to read for this tag. * Data is 0x064dd3, so subnanos prevision is float(0x064dd3) / 2**24 = 0.024625003337860107 ~= 24.6 picoseconds Thus the full timestamp may be interpreted as: 1530056154.707467910024625 6. No more extensions --> end of processing. Note: Packet diagrams created with https://github.com/luismartingarcia/protocol (perhaps with somehand editing) Source for diagrams: mmtrailer; Seconds:32,Nanoseconds:32,Reserved:6,HasExtension:1,FcsValid:1,DeviceID:16,PortID:8,NewFCS:32 mmtlv1; Data:24,Len:2,Final:1,Tag:5 mmSequenceNumber; Reserved:8,Sequence:16,Len:2,Final:1,00000:5 mmFracNano; FractionalNS:24,Len:2,Final:1,00001:5 mmlvl2 Data:32,Tag2:16,len:10,Last:1,11111:5 mmstring ASCII String:64,0000000000000000:16,Len:10,Last:1,11111:5 mmexample DestinationAddress:48,SourceAddress:48,EtherType:16,Payload:144,OrigFCS:32,0x064dd3:24,00:2,1:1,00001:5,0x00:8,0xffb6:16,00:2,0:1,00000:5,0x5b32cdda:32,0x2a2b1a86:32,000000:6,1:1,1:1,0x1dc0:16,0x09:8,NewFCS:32
_______________________________________________ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers