Re: [Wireshark-dev] Adding pcap-ng pipe support to dumpcap
From: Guy Harris on Thursday, 31 August 2017 1:24 PM > On Aug 30, 2017, at 4:58 PM, Stephen Donnelly> wrote: >> At the very least extcap tools should be able to supply data in any format >> understood by wiretap, but since the extcap data currently goes via dumpcap >> (maybe not sensible either?) > > Perhaps not, indeed. > > Currently, there's a protocol between dumpcap and {Wireshark,TShark} allowing > dumpcap to tell *shark "I've appended N more packets to the capture file", to > allow dumpcap to report errors and "here's another capture file" (if it's > doing multiple files), etc.. > > If extcap programs were to speak that protocol when capturing, you could have > the extcap programs behave similarly to dumpcap, writing packets directly to > a file, and have *shark run the extcap program rather than running dumpcap. > I.e., make extcap programs act as substitutes for dumpcap. Agreed. In fact if extcap programs can talk directly to *shark, then dumpcap becomes just another extcap program and not especially privileged. Stephen ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] Adding pcap-ng pipe support to dumpcap
On Aug 30, 2017, at 6:00 PM, Ed Berosetwrote: > One problem is that as dumpcap is currently written, it treats files and > pipes very differently. *Files* and pipes, or *capture devices* and pipes? > but I can't help but think that the general approach you describe is the > better long term strategy. Probably. It means that the interface between *shark and extcap programs would be different - but, while having extcap programs behave like dumpcap might complicate the extcap programs (although some of the code to do that could be in a library used by dumpcap and by extcap programs), it might simplify the Wireshark capture code path. > There are some limitations. Specifically, pipes don't allow random access, > so any file formats that currently require that would need to either be > rewritten Which, for at least one capture file format (Network Monitor format), would be impossible, as we don't define it, Microsoft does (and they're probably not very amenable to changing it, not least because they've deprecated NetMon in favor of Message Analyzer). The only file formats *we* control to any degree are pcap and pcapng, neither of which require random access in order to read them sequentially. If we make extcap programs work like dumpcap, the only pipe involved is the control pipe between *shark and the program - the packets are written directly to a file - but that wouldn't make any difference for those file formats, as you can't, for example, read a NetMon file until it's *completely* written, with a frame table, and you can't do that until you've written all the packets to it. However, I suspect Stephen is thinking of ERF format, which should be writable purely sequentially, so it shouldn't be a problem, whether you're writing to a pipe or to a file that's being read incrementally. ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] Adding pcap-ng pipe support to dumpcap
On Aug 30, 2017, at 4:58 PM, Stephen Donnellywrote: > At the very least extcap tools should be able to supply data in any format > understood by wiretap, but since the extcap data currently goes via dumpcap > (maybe not sensible either?) Perhaps not, indeed. Currently, there's a protocol between dumpcap and {Wireshark,TShark} allowing dumpcap to tell *shark "I've appended N more packets to the capture file", to allow dumpcap to report errors and "here's another capture file" (if it's doing multiple files), etc.. If extcap programs were to speak that protocol when capturing, you could have the extcap programs behave similarly to dumpcap, writing packets directly to a file, and have *shark run the extcap program rather than running dumpcap. I.e., make extcap programs act as substitutes for dumpcap. ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] Adding pcap-ng pipe support to dumpcap
On 08/30/2017 07:58 PM, Stephen Donnelly wrote: Why pcap-ng specifically? Although pcap-ng is higher featured than pcap, it is not Wireshark's internal representation. Pcap-ng is merely the default output format. I don't know about other people's desire for this, but here's mine: I am working on a Raspberry Pi based sniffer tool for IEEE 802.15.4 traffic and want to be able to have the Pi serve up sniffed packets to an instance of Wireshark on another machine. Rather than fiddle with pcap, it seemed to make sense to me to use pcapng as the output format for the Pi-based sniffer. It was only after I had finished that, that I realized that pcapng was not supported via a pipe. Since Wireshark has the ability to detect and read multiple formats already in wiretap, why not leverage that? At the very least extcap tools should be able to supply data in any format understood by wiretap, but since the extcap data currently goes via dumpcap (maybe not sensible either?) they are restricted to pcap only and have to convert to that internally, potentially losing information. Yes, exactly the situation. So I've created a patch which works for my immediate needs, but is rather hack-ish and ugly for any other, at least in part due to some of the factors you mention. Wouldn’t it be better for the capture tool to indicate which of the wiretap formats it intends to use, rather than switching from one fixed format to a different fixed format? This would then support both pcap and pcap-ng intrinsically, as well as all other formats. One problem is that as dumpcap is currently written, it treats files and pipes very differently. Part of that appears to be due to the fact that Windows treats them very differently, but I can't help but think that the general approach you describe is the better long term strategy. There are some limitations. Specifically, pipes don't allow random access, so any file formats that currently require that would need to either be rewritten or omitted for the pipe-able formats. Might be doable in that way, but it would require a lot of re-engineering. Ed ___ Sent via:Wireshark-dev mailing listArchives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] Adding pcap-ng pipe support to dumpcap
> Richard Sharpe Sent: Saturday, 17 June 2017 5:28 AM > > > On Fri, Jun 16, 2017 at 9:36 AM, Kvidera, Evan D> > wrote: > > Hello Wireshark Devs, > > > > My name is Evan Kvidera and I am a senior undergraduate student > > studying Computer Science. I have a decent amount of programming > > experience, but only a little in C. My employer has asked me to try to > > add support for piping pcap-ng captures to Wireshark. > > I have read over the bug report requesting the feature, > > https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=11370. > > > > After reading the mailing list archives here, > > https://www.mail-archive.com/wireshark-dev@wireshark.org/msg6.html > > , it looks like this addition will be nontrivial, but doable, and that > > the changes necessary are all going to be in dumpcap. > > > > I have at least a month or two of full-time work I can dedicate to > > this if necessary, although I am hoping it will not take that long. > > > > I have read through the Wireshark Developer's Guide and looked over > > the style guide for Wireshark. Is there anything else I should know > > before starting development? I will try to develop this as > > independently as possible, but I may have a few questions along the way. > > Hi Evan, > > I looked at this back in 2012 and even proposed a patch that might be useful > to you: > > http://seclists.org/wireshark/2012/May/25 > > No doubt it was a little too simplistic but if I find some time next week > while I am in Seattle I might try to resurrect it and see if it works. Why pcap-ng specifically? Although pcap-ng is higher featured than pcap, it is not Wireshark's internal representation. Pcap-ng is merely the default output format. Since Wireshark has the ability to detect and read multiple formats already in wiretap, why not leverage that? At the very least extcap tools should be able to supply data in any format understood by wiretap, but since the extcap data currently goes via dumpcap (maybe not sensible either?) they are restricted to pcap only and have to convert to that internally, potentially losing information. Wouldn’t it be better for the capture tool to indicate which of the wiretap formats it intends to use, rather than switching from one fixed format to a different fixed format? This would then support both pcap and pcap-ng intrinsically, as well as all other formats. Regards, Stephen ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] Idea about Adding extra functionality in wireshark.
As this sounds like it's more a theoretical thing it might make sense to discuss your ideas (at a high level) on this list first. For instance I'm not sure what Tx power rate means - normally what "Tx power" means to me isn't something Wireshark normally has access to (unless we're talking Wifi). On Wed, Aug 30, 2017 at 11:15 AM, Michael Mann via Wireshark-dev < wireshark-dev@wireshark.org> wrote: > I think the first step is just to submit your patch for review. See > https://wiki.wireshark.org/Development/SubmittingPatches for more details. > > > -Original Message- > From: krishna Kulkarni> To: wireshark-dev > Sent: Tue, Aug 29, 2017 10:20 am > Subject: [Wireshark-dev] Idea about Adding extra functionality in > wireshark. > > Sir >My research paper related to identifying DOS and DDOS attack.research > paper parameter is Tx power rate.Now I want to add this functionality into > wireshrak so the software also identify the attacks defaultly.I want your > team guidance so we develop a better concept in wireshark. we can improve > this product. > ___ > Sent via: Wireshark-dev mailing list > Archives: https://www.wireshark.org/lists/wireshark-dev Unsubscribe: > https://www.wireshark.org/mailman/options/wireshark-dev > mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe > > > > ___ > Sent via:Wireshark-dev mailing list > Archives:https://www.wireshark.org/lists/wireshark-dev > Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev > mailto:wireshark-dev-requ...@wireshark.org?subject= > unsubscribe > ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] Idea about Adding extra functionality in wireshark.
I think the first step is just to submit your patch for review. See https://wiki.wireshark.org/Development/SubmittingPatches for more details. -Original Message- From: krishna KulkarniTo: wireshark-dev Sent: Tue, Aug 29, 2017 10:20 am Subject: [Wireshark-dev] Idea about Adding extra functionality in wireshark. Sir My research paper related to identifying DOS and DDOS attack.research paper parameter is Tx power rate.Now I want to add this functionality into wireshrak so the software also identify the attacks defaultly.I want your team guidance so we develop a better concept in wireshark. we can improve this product. ___Sent via:Wireshark-dev mailing list Archives: https://www.wireshark.org/lists/wireshark-devUnsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] [RFC] Vendor-specific dissector extension for EtherNet/IP
My mail client decided to butcher the formatting of your email, so I manually tried to reformat it so that my inline replies make sense. Hopefully the mail client doesn't butcher it on the way out too. Sorry for the inconvience. Le 29/08/2017 à 21:34, Michael Mann via Wireshark-dev a écrit : >> The answer depends on exactly what you are trying to do, some things >> will be easier than others. >> 1. If you want to add vendor specific objects, that can easily be done >> in Lua because there is a dissector table that you can just register >> your vendor specific class with ("cip.class.iface"). There should be >> numerous examples of Lua using a dissector table (just not specifically >> for CIP). > I have to admit the way dissector tables work is not clear for me. There are 2 types of dissector tables, one for heuristics, one for fixed values. For the "fixed" values one, you associate a value with a dissection function. Using "cip.class.iface" as an example, you are associating the numeric value of your object class with the dissector table, so the CIP dissector can use that table to decide how/what to dissect next. With a heuristic dissection table, you just register a "heuristic" function with the dissector table. Within that function could can decide if the "packet" should be handled by your dissector. I mention both because I forgot that there is a heuristic dissector for service codes ("cip.sc"). It looks like you need a combination of a dissector function for your vendor-specific class as well as a heuristic function to handle the common services within your vendor specific class. You can mimic the behavior of the Connection Configuration Object dissection for your vendor-specific objects. It first registers a "(CIP) object dissection function" with: cip_class_cco_handle = create_dissector_handle( dissect_cip_class_cco, proto_cip_class_cco ); dissector_add_uint( "cip.class.iface", CI_CLS_CCO, cip_class_cco_handle ); But it also adds a huerisitic function: heur_dissector_add("cip.sc", dissect_class_cco_heur, "CIP Connection Configuration Object", "cco_cip", proto_cip_class_cco, HEURISTIC_ENABLE); The dissect_class_cco_heur() basically checks the EPATH for the Connection Configuration Object class value and then calls the main object dissection function, dissect_cip_class_cco. > I registered my dissector for a vendor-specific class, and I observed the > following > behaviour: - When the service is common (Get_Attribute_List for example), the > dissector > is not called. > - when the service is vendor-specific, the dissector is then called. > So I can implement the vendor-specific services for vendor-specific classes > at least, > with a new dissector. > But I don't understand why discriminate dissector in the class dissector > table based > on the service being known or not, any idea why it was implemented this way? > Since the > service is parsed and interpreted before the path (so the class), > wouldn't > it make more sense to actually make a dissector table based on the service? Hopefully the explanation above satisfies your questions. >> 2. There is no support currently for "classless" service codes (like >> those used in Rockwell Automation PLCs), which is what >> _https://www.wireshark.org/lists/ethereal-dev/200601/msg00174.html_ appears >> to be talking about. > As I understand it the service codes mentioned in that thread are class > specific. > I have never encountered "classless" service codes until now, I didn't even > know that > existed (as CIP doesn't implement this behaviour, or at least I couldn't find > it in the documentation). To me, in layman's terms, the format of a CIP message is In the CIP specs, they focus on being an object class and instance, but for some Rockwell PLCs, the is just a string (and there is no class in the message). That's what I mean when I refer to "classless" requests. >> 2. If you want to add vendor specific services to already supported >> objects, that would be more difficult to do in Lua because there isn't a >> dissector table hook for them. I'm not sure there would be a way to >> handle the "general" case of registering service + class into a >> dissector table, but you could add dissector tables (patching >> packet-cip.c) for specific objects (Identity, ConnectionManager, etc) >> and submit just that part as a patch for inclusion in base Wireshark code. This is where I started to steer you incorrectly. The heuristic functionality should be able to handle this case without issue. >> 3. Vendor specific attributes of an object would have the same >> difficulty in Lua and would need dissector tables. > Well, there is the cip.data_segment.iface dissector table, but then one would > need to > check (in every dissector register in this table) the class code, and also the > instance id (to determine if it is a class attribute or an instance > attribute). > I'm not sure how this dissector table is used
Re: [Wireshark-dev] Adding pcap-ng pipe support to dumpcap
On Wed, Aug 30, 2017 at 1:21 AM, Anders Bromanwrote: > > > -Original Message- > From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of > Ed Beroset > Sent: den 29 augusti 2017 21:40 > To: wireshark-dev@wireshark.org > Subject: Re: [Wireshark-dev] Adding pcap-ng pipe support to dumpcap > > On 08/29/2017 02:35 PM, Richard Sharpe wrote: >>> On Tue, Aug 29, 2017 at 10:50 AM, Ed Beroset wrote: On 06/16/2017 01:27 PM, Richard Sharpe wrote: I've just encountered a need for this as well. Have you made progress, Evan? Do you want some help? >>> >>> Evan seems to have dropped off the radar. I outlined to Evan an >>> approach for doing this, so I could send it to Ed instead ... >> >>It would be helpful if you could. I've looked at your old patch and I've >>been looking over the dumpcap.c code and have been making some notes, but it >>would nice to have >additional input on this, especially since you've already >>looked at it. Thanks! FWIW, here are the notes I sent Evan and Ed: --- > The patch that I pointed you at is incomplete because it does not > handle PCAPNG captures being piped at you. In addition I am no longer > happy with exposing the internals of the struct capture_src like that. > > It should be easy to add pcapng support by detecting in > cap_pipe_open_live that you are dealing with PCAPNG rather than PCAP, > since you will see that you have an PCAPNG capture because the Section > Header Block magic (block type) is 32 bits and is different than any > of the PCAP magic values, and there must be at least one SHB in a > PCAPNG capture. > > Thus you can use that to choose which state machine should be used for > the subsequent reads. > > I think it might be cleaner to extract the current PCAP state machine > into its own function and create a new one for PCAPNG ... and I will > have more thoughts later after I have thought about this some more. More thoughts ... 1. Rename cap_pipe_dispatch to pcap_pipe_dispatch. 2. Create a new pcapng_pipe_dispatch. It only has to deal with two record types. SHBs and all others. The reason you have to deal with SHBs specially is that they can change the endianness of the capture until you see the next SHB. Each block, including the SHB has a length field as the second 32-bit field and the last 32-bit field. The SHB's length cannot be interpreted until you figure out the endianness which is determined from the third 32-bit field. It also contains a total length for the section. 3. The first SHB has to be dealt with specially since you need to read the first 32-bits (the MAGIC) to figure out whether or not you are dealing with PCAP or PCAPNG. After that, SHBs still need to be dealt with specially, but other blocks are simple to deal with. Read the first 8 bytes, and if the MAGIC says it is not an SHB, the second 4-byte field tells you how many more bytes to read. Then you can simply write the lot out. For each non-first SHB, you need to read another 4 bytes to figure out the endianness field (4-bytes) to determine how to interpret the second field. 4. Modify pcap_dispatch to call pcap_pipe_dispatch or pcapng_pipe_dispatch depending on the the type of the capture determined from reading the very first MAGIC field. 5. cap_pipe_dispatch may look complex because of all those #ifdef's but that is simply to support Windows as well as Linux. It should be possible to create pcapng_pipe_dispatch along similar lines as well. There is probably only two states, or maybe three states, for pcapng_pipe_dispatch: PCAPNG_STATE_EXPECT_PREAMBLE (for the first 8 bytes), PCAPNG_NG_STATE_READ_DATA for the remaining data in the record, and PCAPNG_STATE_EXPECT_SHB_ENDIAN for the 4-byte endianness indicator of an SHB. -- Regards, Richard Sharpe (何以解憂?唯有杜康。--曹操) ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] [RFC] Vendor-specific dissector extension for EtherNet/IP
Le 29/08/2017 à 21:34, Michael Mann via Wireshark-dev a écrit : The answer depends on exactly what you are trying to do, some things will be easier than others. 1. If you want to add vendor specific objects, that can easily be done in Lua because there is a dissector table that you can just register your vendor specific class with ("cip.class.iface"). There should be numerous examples of Lua using a dissector table (just not specifically for CIP). I have to admit the way dissector tables work is not clear for me. I registered my dissector for a vendor-specific class, and I observed the following behaviour: - When the service is common (Get_Attribute_List for example), the dissector is not called. - when the service is vendor-specific, the dissector is then called. So I can implement the vendor-specific services for vendor-specific classes at least, with a new dissector. But I don't understand why discriminate dissector in the class dissector table based on the service being known or not, any idea why it was implemented this way? Since the service is parsed and interpreted before the path (so the class), wouldn't it make more sense to actually make a dissector table based on the service? 2. There is no support currently for "classless" service codes (like those used in Rockwell Automation PLCs), which is what _https://www.wireshark.org/lists/ethereal-dev/200601/msg00174.html_ appears to be talking about. As I understand it the service codes mentioned in that thread are class specific. I have never encountered "classless" service codes until now, I didn't even know that existed (as CIP doesn't implement this behaviour, or at least I couldn't find it in the documentation). 2. If you want to add vendor specific services to already supported objects, that would be more difficult to do in Lua because there isn't a dissector table hook for them. I'm not sure there would be a way to handle the "general" case of registering service + class into a dissector table, but you could add dissector tables (patching packet-cip.c) for specific objects (Identity, ConnectionManager, etc) and submit just that part as a patch for inclusion in base Wireshark code. Re 3. Vendor specific attributes of an object would have the same difficulty in Lua and would need dissector tables. Well, there is the cip.data_segment.iface dissector table, but then one would need to check (in every dissector register in this table) the class code, and also the instance id (to determine if it is a class attribute or an instance attribute). I'm not sure how this dissector table is used though: is it only for attribute data or also service data? 4. I believe Lua will "override" any value registered to a dissector table, so you could write the "vendor specific" portion, for say the Identity object, but then you'd have to duplicate all of the dissection currently being done for it in your Lua script. I did test with setting a lua dissector for Identity in the cip.class.iface, and on packets with common services it wasn't triggered (I didn't have packets with vendor-specific services call for Identity). So apparently it does not override the default dissector with the lua one (at least with a common service). 5. Also note that not all "open" objects are supported in packet-cip.c. It would be appreciated that if you added dissection for any of those, that you provide a patch for integration here: https://code.wireshark.org/review (see https://wiki.wireshark.org/Development/SubmittingPatches for more details). If you're more familiar with Lua than C, you can put the Lua script here: https://wiki.wireshark.org/Contrib, but I'd probably end up taking it and converting it to C. If I encounter those I can implement it (in C or lua, both are fine for me), but it is not my priority for now. The problem I see here is that the system of dissector table does not fit our needs in this context. Ideally I would see one custom dissector just for the service and path (so service, classes and instances), and one for the payload (attributes and services data). All the structure would be made in C, and just small portions of lua would need to be used to (dynamically) provide names for classes, services and attributes. Then, a bigger lua script would provide a full dissector for the payload (attributes or services data), based on the previously parsed informations. How I have no idea if this is possible in wireshark, any thoughts? ___ Sent via:Wireshark-dev mailing listArchives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] Adding pcap-ng pipe support to dumpcap
-Original Message- From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of Ed Beroset Sent: den 29 augusti 2017 21:40 To: wireshark-dev@wireshark.org Subject: Re: [Wireshark-dev] Adding pcap-ng pipe support to dumpcap On 08/29/2017 02:35 PM, Richard Sharpe wrote: >> On Tue, Aug 29, 2017 at 10:50 AM, Ed Berosetwrote: >>> On 06/16/2017 01:27 PM, Richard Sharpe wrote: >>> >>> I've just encountered a need for this as well. Have you made >>> progress, Evan? Do you want some help? >> >> Evan seems to have dropped off the radar. I outlined to Evan an >> approach for doing this, so I could send it to Ed instead ... > >It would be helpful if you could. I've looked at your old patch and I've been >looking over the dumpcap.c code and have been making some notes, but it would >nice to have >additional input on this, especially since you've already looked >at it. Thanks! > >Ed Please send the notes to the list it may be useful for future reference. Regards Anders ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe