Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types
Hi, Can I register two new PCAP-NG block codes? I’d like: • 0x54534442 – Text Source Descriptor Block (TSDB) • 0x54524200 – Text Source Record Block (TRB) What do I need to do to register these? See below for the reason for request. Thanks and regards…Paul -Original Message- From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of Guy Harris Sent: 19 June 2017 22:54 To: Developer support list for Wireshark Subject: Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types On Jun 19, 2017, at 3:06 AM, Paul Offord wrote: >> Do you mean "deal with arbitrary CSV and TSV files" or "deal with IIS Web >> logs that are in CSV and TSV format"? > > Yes - I mean arbitrary comma, tab and space separated. I think Microsoft have an Excellent program that reads CSV files - and possibly TSV and SSV as well - and lets you do various forms of processing on them; it's available for Windows and macOS. I think there's another program called LibreOffice Calc that's available for those platforms and for Linux, *BSD, and other OSes as well. I.e., what does *Wireshark* provide here, for the *general* case of CSV/TSV/SSV files, that a spreadsheet program doesn't? > Going back to the main objective, what I'm proposing is that we enable a > PCAP-NG file to contain any text-based data together with a schema - JSON or > XML or PCAP-NG proprietary. There would be one WTAP_ENCAP_xxx type for any > text-based data type. Decoding would be based on the schema data. pcapng, unlike pcap, can have blocks in it that *don't* correspond to packets. What's needed here are new pcapng block types: a "text source description block" (TSDB), which contains a schema and options (such as comments); a "text record block" (TRB), which contains: a "text source ID", which indicates which TSDB corresponds to the source of the record (the first TSDB has an ID of 0, the next one an ID of 1, etc., along the lines of Interface Description Blocks); the text; options (such as comments). This would *not* require a DLT_/LINKTYPE_ or a WTAP_ENCAP_ value. It would also allow packets and text records to be combined into one file. ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe __ This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour House, Coopers End Lane, Stansted, Essex CM24 1SJ __ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com __ ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types
Hi, Sorry - Slight amendment. Can I register two new PCAP-NG block codes? I’d like: • 0x42445354 – Text Source Descriptor Block (TSDB) • 0x00425254 – Text Source Record Block (TRB) What do I need to do to register these? See below for the reason for request. Thanks and regards…Paul -Original Message- From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of Paul Offord Sent: 25 July 2017 10:26 To: Developer support list for Wireshark Subject: Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types Hi, Can I register two new PCAP-NG block codes? I’d like: • 0x54534442 – Text Source Descriptor Block (TSDB) • 0x54524200 – Text Source Record Block (TRB) What do I need to do to register these? See below for the reason for request. Thanks and regards…Paul -Original Message- From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of Guy Harris Sent: 19 June 2017 22:54 To: Developer support list for Wireshark Subject: Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types On Jun 19, 2017, at 3:06 AM, Paul Offord wrote: >> Do you mean "deal with arbitrary CSV and TSV files" or "deal with IIS Web >> logs that are in CSV and TSV format"? > > Yes - I mean arbitrary comma, tab and space separated. I think Microsoft have an Excellent program that reads CSV files - and possibly TSV and SSV as well - and lets you do various forms of processing on them; it's available for Windows and macOS. I think there's another program called LibreOffice Calc that's available for those platforms and for Linux, *BSD, and other OSes as well. I.e., what does *Wireshark* provide here, for the *general* case of CSV/TSV/SSV files, that a spreadsheet program doesn't? > Going back to the main objective, what I'm proposing is that we enable a > PCAP-NG file to contain any text-based data together with a schema - JSON or > XML or PCAP-NG proprietary. There would be one WTAP_ENCAP_xxx type for any > text-based data type. Decoding would be based on the schema data. pcapng, unlike pcap, can have blocks in it that *don't* correspond to packets. What's needed here are new pcapng block types: a "text source description block" (TSDB), which contains a schema and options (such as comments); a "text record block" (TRB), which contains: a "text source ID", which indicates which TSDB corresponds to the source of the record (the first TSDB has an ID of 0, the next one an ID of 1, etc., along the lines of Interface Description Blocks); the text; options (such as comments). This would *not* require a DLT_/LINKTYPE_ or a WTAP_ENCAP_ value. It would also allow packets and text records to be combined into one file. ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe __ This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour House, Coopers End Lane, Stansted, Essex CM24 1SJ __ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com __ ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe __ This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminat
Re: [Wireshark-dev] Wireshark support for Qt4.8
On 25 July 2017 at 06:31, Michał Łabędzki wrote: > +1 for drop Qt4.8 (about 6 years old lib... enough time to upgrade it > https://en.wikipedia.org/wiki/Qt_version_history) > > What about new Qt5 version? Qt5.2 to support Ubuntu 14.04 LTS? > (however it seems to be not a problem to have newer one > https://wiki.qt.io/Install_Qt_5_on_Ubuntu ) > > CentOS? http://wiki.qt.io/How_to_Install_Qt_5_and_Qwt_on_CentOS_6 > Maybe something like Docker saves the day? > > One thing about dropping practise - it is good to write about that to > some file "requirements.txt?" It may greatly help some people to port > or get last available Wireshark for theirs environment. For example: > "4th July 2018 - drop Qt 4.8 support, git sha: 4ff4ff4ff4..." > > We do record this kind of information already, see the lifecycle page on the wiki, in the "End of life planning" section: https://wiki.wireshark.org/Development/LifeCycle > PS. Traditionally +1 for drop others build systems (autotools?). [but, > +1 also for drop cmake and keeps only autotools...] > > 2017-07-24 16:06 GMT+02:00 Alexis La Goutte : > > > > > > On Mon, Jul 24, 2017 at 3:34 PM, Roland Knall wrote: > >> > >> Hi > >> > >> Wireshark master currently does not build with Qt 4.8. The reason for > >> that, can be seen in > >> > >> https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=13909 > >> > >> These issues should be fixed in the near future. But it raises the > bigger > >> topic, if Qt 4.8 should be supported in whatever version comes after > 2.4. > >> Are there any big reasons or support cases, where Qt 4.8 is a definite > >> requirement and going to Qt 5.x is not an option? > >> > >> The list on > >> https://wiki.wireshark.org/Development/Support_library_version_tracking > is a > >> starting point for discussion I would assume, but as we no longer > support Qt > >> 4.6, SLES 12 and RHEL/CentoOS 7 would be my point of reference, and both > >> support Qt 5. > >> > >> Am I missing some references here? An can we start to discuss moving the > >> Qt version to 5.x > >> > > +1 for may be drop 4.x support > > if i remember there is some -(core)dev using some old (CentOS?) linux > system > > and don't have Qt 5 support... > > > > > > > -- Graham Bloice ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
[Wireshark-dev] Wireshark 2.5.0 ws.css errors in docbook\developer_guide_chm.vcxproj
ws.css has errors on lines 107 (body pre) and 116 (tt.literal, code.literal where color 333 is written, but color code 333 is not a valid color value. -- Jay Turner Contact: 214-587-7364 ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
[Wireshark-dev] Hierarchy of fields & offsets
Hi, Looking at some of the parsed data in my trials, I am seeing odd things such as : format is : [ftenum] [offset] [name or abbrev] ([length]) : FT_PROTOCOL 66 mysql(9) : FT_UINT24 66 mysql.packet_length(3) : 5 FT_UINT8 69 mysql.packet_number(1) : 0 FT_NONE 70 mysql.request(1) : 02 FT_UINT8 70 mysql.command(1) : 2 FT_STRING 71 mysql.schema(4) : test Notice how mysql.command and mysql.schema are in the hierarchy children of mysql.request, however mysql.request's length is only 1 byte. FT_BYTES 198 smb2.security_blob(120) : 60:76:06:06:2b:06:01:05:05:02:a0:6c:30:6a:a0:3c:30:3a:06:0a:2b:06:01:04:01:82:37:02:02:1e:06:09:2a:86:48:82:f7:12:01:02:02:06:09:2a:86:48:86:f7:1 2:01:02:02:06:0a:2a:86:48:86:f7:12:01:02:02:03:06:0a:2b:06:01:04:01:82:37:02:02:0a:a3:2a:30:28:a0:26:1b:24:6e:6f:74:5f:64:65:66:69:6e:65:64:5f:69:6e:5f:52:46:43:34:31:37:38:40:70:6c:65:61:73:65:5f:69:6 7:6e:6f:72:65 FT_UINT32 190 smb2.olb.offset(2) : 0x0080 FT_UINT32 192 smb2.olb.length(2) : 120 FT_PROTOCOL 198 gss-api(120) : Notice how smb2.olb.offset & smb2.olb.length are under smb2.security_blob, but their offset starts at 190/192 while smb2.security_blob starts at 198 This is extremely confusing to say the least and it makes it very hard to interpret the data in an automated manner. Any reason why this is done in this way ? I would personally expect a "parent" field to have offset/length that are consistent with its children. Thanks, Hassan ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] Hierarchy of fields & offsets
On Jul 25, 2017, at 3:26 PM, Sultan, Hassan via Wireshark-dev wrote: > Any reason why this is done in this way? I don't know, but, whatever it is, it's not a *good* reason. Perhaps they didn't know how to handle a request whose length isn't known until you finish dissecting it. The answer is "give it an initial length of -1, to cover the rest of the data, and then set the length at the end"; I've changed the MySQL dissector in the master and 2.4 branches to do that. ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] "[UNVERIFIED SENDER]Re: Hierarchy of fields & offsets
Awesome, thanks ! So shall I assume that whenever I detect something of the kind, it's an issue that needs resolved ? If that's the case I'll be more than happy to add detection for this in my code and run a bunch of captures through it to detect them all (or at least as many as the captures allow me to detect). Also, is the smb2 case a bug as well ? Thx, Hassan -Original Message- From: Guy Harris [mailto:g...@alum.mit.edu] Sent: Tuesday, July 25, 2017 3:45 PM To: Developer support list for Wireshark Cc: Sultan, Hassan Subject: "[UNVERIFIED SENDER]Re: [Wireshark-dev] Hierarchy of fields & offsets On Jul 25, 2017, at 3:26 PM, Sultan, Hassan via Wireshark-dev wrote: > Any reason why this is done in this way? I don't know, but, whatever it is, it's not a *good* reason. Perhaps they didn't know how to handle a request whose length isn't known until you finish dissecting it. The answer is "give it an initial length of -1, to cover the rest of the data, and then set the length at the end"; I've changed the MySQL dissector in the master and 2.4 branches to do that. ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] Hierarchy of fields & offsets
On Jul 25, 2017, at 3:26 PM, Sultan, Hassan via Wireshark-dev wrote: > FT_BYTES 198 smb2.security_blob(120) : > 60:76:06:06:2b:06:01:05:05:02:a0:6c:30:6a:a0:3c:30:3a:06:0a:2b:06:01:04:01:82:37:02:02:1e:06:09:2a:86:48:82:f7:12:01:02:02:06:09:2a:86:48:86:f7:1 > 2:01:02:02:06:0a:2a:86:48:86:f7:12:01:02:02:03:06:0a:2b:06:01:04:01:82:37:02:02:0a:a3:2a:30:28:a0:26:1b:24:6e:6f:74:5f:64:65:66:69:6e:65:64:5f:69:6e:5f:52:46:43:34:31:37:38:40:70:6c:65:61:73:65:5f:69:6 > 7:6e:6f:72:65 > FT_UINT32 190 > smb2.olb.offset(2) : 0x0080 > FT_UINT32 192 > smb2.olb.length(2) : 120 > FT_PROTOCOL 198 gss-api(120) : > > Notice how smb2.olb.offset & smb2.olb.length are under smb2.security_blob, > but their offset starts at 190/192 while smb2.security_blob starts at 198 Now we're not just dealing with data structures, we're dealing with pointers If the pointer and length are considered part of the "security blob", then the "security blob" needs to have three subfields - offset, length, and content - and therefore must be an FT_NONE and must not be a contiguous blob. That would require that Wireshark support fields with more than one offset/length pair; if you clicked on them, they'd highlight more than one region of the hex pane. If the pointer and length are *not* considered part of the security blob, then they shouldn't be under smb2.security_blob. ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] "[UNVERIFIED SENDER]Re: Hierarchy of fields & offsets
> -Original Message- > From: Guy Harris [mailto:g...@alum.mit.edu] > Sent: Tuesday, July 25, 2017 5:38 PM > To: Developer support list for Wireshark > Cc: Sultan, Hassan > Subject: "[UNVERIFIED SENDER]Re: [Wireshark-dev] Hierarchy of fields & > offsets > > On Jul 25, 2017, at 3:26 PM, Sultan, Hassan via Wireshark-dev d...@wireshark.org> wrote: > > > FT_BYTES 198 smb2.security_blob(120) : > 60:76:06:06:2b:06:01:05:05:02:a0:6c:30:6a:a0:3c:30:3a:06:0a:2b:06:01:04:01:8 > 2:37:02:02:1e:06:09:2a:86:48:82:f7:12:01:02:02:06:09:2a:86:48:86:f7:1 > > > 2:01:02:02:06:0a:2a:86:48:86:f7:12:01:02:02:03:06:0a:2b:06:01:04:01:82:37:02: > 02:0a:a3:2a:30:28:a0:26:1b:24:6e:6f:74:5f:64:65:66:69:6e:65:64:5f:69:6e:5f:52: > 46:43:34:31:37:38:40:70:6c:65:61:73:65:5f:69:6 > > 7:6e:6f:72:65 > > FT_UINT32 190 > > smb2.olb.offset(2) : 0x0080 > > FT_UINT32 192 > > smb2.olb.length(2) : 120 > > FT_PROTOCOL 198 > > gss-api(120) : > > > > Notice how smb2.olb.offset & smb2.olb.length are under smb2.security_blob, > but their offset starts at 190/192 while smb2.security_blob starts at 198 > > Now we're not just dealing with data structures, we're dealing with > pointers > > If the pointer and length are considered part of the "security blob", then the > "security blob" needs to have three subfields - offset, length, and content - > and > therefore must be an FT_NONE and must not be a contiguous blob. That would > require that Wireshark support fields with more than one offset/length pair; > if > you clicked on them, they'd highlight more than one region of the hex pane. I'm not sure I understand what you mean here (likely due to me still ramping up on the code). Are you saying that, assuming the blob offset/length fields are part of the blob, using FT_BYTES for the blob here is wrong b/c the 3 fields under it are not within the offset/length defined for the blob, and instead FT_NONE should be used ? Should I understand from this that FT_NONE is purely an abstract grouping without any meaning as to position/length of the fields under it ? > If the pointer and length are *not* considered part of the security blob, then > they shouldn't be under smb2.security_blob. Not sure what the authoritative source would be for that, but looking at the parser in Network Monitor 3 it has them outside the blob : UINT16 SecurityBufferOffset; UINT16 SecurityBufferLength; UINT32 Reserved2; switch { case CurrentProtocolOffset < SecurityBufferOffset: BLOB(SecurityBufferOffset - CurrentProtocolOffset) BufferPadding; } SMBSecurityBlob(SecurityBufferLength) securityBlob Thanks, Hassan ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] "[UNVERIFIED SENDER]Re: Hierarchy of fields & offsets
On Jul 25, 2017, at 5:58 PM, Sultan, Hassan wrote: > I'm not sure I understand what you mean here (likely due to me still ramping > up on the code). > > Are you saying that, assuming the blob offset/length fields are part of the > blob, using FT_BYTES for the blob here is wrong b/c the 3 fields under it are > not within the offset/length defined for the blob, and instead FT_NONE should > be used ? Should I understand from this that FT_NONE is purely an abstract > grouping without any meaning as to position/length of the fields under it ? I'm saying that there are probably people who think of the length and offset as part of the security information, along with the blob to which they refer, and people who think of the length and offset as separate from the blob to which they refer, so there's not an "obvious" way to represent them in the protocol tree ("obvious" in the sense that "nobody would make a good argument for doing it differently"). For people in the former group, the right way to do the field would be as an FT_NONE, with the three items underneath it, and with the FT_NONE item being composed of two disconnected ranges. The blob data itself could just be dissected as NTLMSSP or GSSAPI, without an FT_BYTES field; the top-level protocol item should have all the blob's data in it. For people in the latter group, the right way to do it would be to have separate fields for the length and offset, not under the item for the security blob, with the security blob as a separate item - which, again, could just be NTLMSSP or GSSAPI, without an FT_BYTES field. >> If the pointer and length are *not* considered part of the security blob, >> then >> they shouldn't be under smb2.security_blob. > > Not sure what the authoritative source would be for that, but looking at the > parser in Network Monitor 3 it has them outside the blob : > > UINT16 SecurityBufferOffset; > UINT16 SecurityBufferLength; > UINT32 Reserved2; > switch > { > case CurrentProtocolOffset < SecurityBufferOffset: > BLOB(SecurityBufferOffset - CurrentProtocolOffset) > BufferPadding; > } > SMBSecurityBlob(SecurityBufferLength) securityBlob So NetMon treats them as separate from the item to which they refer, i.e. the second way. Whoever wrote the Wireshark code chose something like the first way, but I don't think it's done correctly. ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe