Re: External USB fuzzing
On Wed, 6 Jun 2018, Andrey Konovalov wrote: > On Tue, Jun 5, 2018 at 10:06 PM, Alan Stern wrote: > > On Tue, 5 Jun 2018, Andrey Konovalov wrote: > > [...] > > > What do you need that's different from what gadgetfs provides? > > 1. It does some sanity checks on the provided USB descriptors, which > we don't really want, since providing improper descriptors is a way to > find bugs. > > 2. Replies to some USB requests without asking the user. Same thing, > giving the user (== syzkaller) to reply something improper can trigger > some bugs. > > 3. File based interface is somewhat inconvenient for integration with > syzkaller. > > (4. Doesn't support SuperSpeed and doesn't support multiple UDC > devices. This can be fixed of course. ) > > Basically what (I think) I want is an ioctl-based GadgetFS-like > interface, that doesn't implement any USB protocol logic, but simply > passes all received requests to the user and asks for replies. I see. Yes, hacking the gadgetfs driver is going to be the quickest way to get that sort of interface. > >> Could you elaborate on this? AFAIU the dummy device contains both a > >> virtual UDC and a virtual HCD that are connected to each other and > >> allow to plug a gadget device into usb core within the kernel. How do > >> you test a driver for an actual host controller? Is there a way to > >> connect a dummy UDC with an actual HCD? > > > > You can't connect a dummy UDC to an actual HCD. But you can connect an > > actual UDC to an actual HCD -- people do it all the time. > > > > Normally the UDC and the HCD reside on separate computers. For > > example, mobile devices typically have UDCs, and you can connect them > > to PCs. But there is also UDC hardware available for Linux on Intel > > systems, so you can have the UDC and the HCD in the same computer if > > that's what you want. Or two separate PCs, both running Linux. Or > > even a Linux PC gadget connected to a Windows or OS X host! > > I see, but that all requires real hardware, right? You can't just run > it in a VM (that's what I'm planning to do with dummy_hcd)? Not unless the hypervisor has virtualized the UDC (which I doubt any of them do.) Still, it might be worthwhile running some tests on bare metal, even if that means only doing one test at a time. > [...] > > > Right. For example, usb-storage uses a work-queue routine for SCSI > > scanning when a new device is probed. And in that routine, the SCSI > > layer often delegates some of the scanning work to async helper > > threads, depending on the how the system is configured. > > Could you point me to the code that does that? I mean SCSI scanning. SCSI scanning is done by several routines in drivers/scsi/scsi_scan.c. The initial entry point is scsi_scan_host(). > [...] > > >> 2. All the virtual UDCs get the same name "dummy_udc". And since the > >> process of UDC lookup to bind a gadget driver is based on the UDC name > >> (see usb_gadget_probe_driver()), we can't bind different gadget > >> drivers to different UDCs (and therefore connect to different hubs). > > > > That can be changed pretty easily. For virtual UDCs beyond the first, > > we could append "-2", "-3", and so on to the "dummy_udc" name, for > > example. > > I wrote a patch to implement this and then realized that it's actually > not needed. Even though all UDC devices have the same name > "dummy_udc", it seems they are actually renamed by someone and end up > getting names "dummy_udc.0", "dummy_udc.1" and so on (that'w what I > see when I add printk to usb_gadget_probe_driver()). Probably the platform driver core does this. > What initially confused me was that gadgetfs always binds to the same > UDC, but that's because it doesn't set the udc_name field of the > usb_gadget_driver struct (and also because it always expects the UDC > name to be the one returned by usb_get_gadget_udc_name(), but that > part I patched when I was trying to bind it to some other UDC). If a gadget driver don't specify the udc_name then it binds to the first available UDC. So if you load some other gadget driver (such as g_zero) before gadgetfs, then gadgetfs would bind to the second UDC. Alan Stern -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: External USB fuzzing
On Tue, Jun 5, 2018 at 10:06 PM, Alan Stern wrote: > On Tue, 5 Jun 2018, Andrey Konovalov wrote: [...] > What do you need that's different from what gadgetfs provides? 1. It does some sanity checks on the provided USB descriptors, which we don't really want, since providing improper descriptors is a way to find bugs. 2. Replies to some USB requests without asking the user. Same thing, giving the user (== syzkaller) to reply something improper can trigger some bugs. 3. File based interface is somewhat inconvenient for integration with syzkaller. (4. Doesn't support SuperSpeed and doesn't support multiple UDC devices. This can be fixed of course. ) Basically what (I think) I want is an ioctl-based GadgetFS-like interface, that doesn't implement any USB protocol logic, but simply passes all received requests to the user and asks for replies. [...] >> Could you elaborate on this? AFAIU the dummy device contains both a >> virtual UDC and a virtual HCD that are connected to each other and >> allow to plug a gadget device into usb core within the kernel. How do >> you test a driver for an actual host controller? Is there a way to >> connect a dummy UDC with an actual HCD? > > You can't connect a dummy UDC to an actual HCD. But you can connect an > actual UDC to an actual HCD -- people do it all the time. > > Normally the UDC and the HCD reside on separate computers. For > example, mobile devices typically have UDCs, and you can connect them > to PCs. But there is also UDC hardware available for Linux on Intel > systems, so you can have the UDC and the HCD in the same computer if > that's what you want. Or two separate PCs, both running Linux. Or > even a Linux PC gadget connected to a Windows or OS X host! I see, but that all requires real hardware, right? You can't just run it in a VM (that's what I'm planning to do with dummy_hcd)? [...] > Right. For example, usb-storage uses a work-queue routine for SCSI > scanning when a new device is probed. And in that routine, the SCSI > layer often delegates some of the scanning work to async helper > threads, depending on the how the system is configured. Could you point me to the code that does that? I mean SCSI scanning. [...] >> 2. All the virtual UDCs get the same name "dummy_udc". And since the >> process of UDC lookup to bind a gadget driver is based on the UDC name >> (see usb_gadget_probe_driver()), we can't bind different gadget >> drivers to different UDCs (and therefore connect to different hubs). > > That can be changed pretty easily. For virtual UDCs beyond the first, > we could append "-2", "-3", and so on to the "dummy_udc" name, for > example. I wrote a patch to implement this and then realized that it's actually not needed. Even though all UDC devices have the same name "dummy_udc", it seems they are actually renamed by someone and end up getting names "dummy_udc.0", "dummy_udc.1" and so on (that'w what I see when I add printk to usb_gadget_probe_driver()). What initially confused me was that gadgetfs always binds to the same UDC, but that's because it doesn't set the udc_name field of the usb_gadget_driver struct (and also because it always expects the UDC name to be the one returned by usb_get_gadget_udc_name(), but that part I patched when I was trying to bind it to some other UDC). Thanks! -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: External USB fuzzing
On Tue, 5 Jun 2018, Andrey Konovalov wrote: > On Mon, Jun 4, 2018 at 9:37 PM, Alan Stern wrote: > > On Mon, 4 Jun 2018, Andrey Konovalov wrote: > > Hi, Alan! Hi! > [...] > > >> The perfect solution would be to have something like /dev/tun for USB, > >> where you can write USB packets and the kernel would synchronously > >> process them. I'm not sure whether this is possible (highly > >> asynchronous USB core subsystem architecture + all communication is > >> initiated by the host). > > > > gadgetfs is a little like that already, except of course that the > > processing is not synchronous. > > Yes, I started with GadgetFS initially, but actually ended up writing > my own interface that provides userspace interface to the gadget API > instead. Once I figure out what exactly I need from it, it might be a > good idea to merge it with GadgetFS as some special mode. Or not, not > sure yet. What do you need that's different from what gadgetfs provides? > > Another aspect of this would be fuzzing USB host controller drivers. > > In theory you could start doing this now, although you would have to > > use a real UDC instead of dummy-hcd. However, this may create more > > complications that you want to deal with at the moment. > > Could you elaborate on this? AFAIU the dummy device contains both a > virtual UDC and a virtual HCD that are connected to each other and > allow to plug a gadget device into usb core within the kernel. How do > you test a driver for an actual host controller? Is there a way to > connect a dummy UDC with an actual HCD? You can't connect a dummy UDC to an actual HCD. But you can connect an actual UDC to an actual HCD -- people do it all the time. Normally the UDC and the HCD reside on separate computers. For example, mobile devices typically have UDCs, and you can connect them to PCs. But there is also UDC hardware available for Linux on Intel systems, so you can have the UDC and the HCD in the same computer if that's what you want. Or two separate PCs, both running Linux. Or even a Linux PC gadget connected to a Windows or OS X host! ... > The main question is: suppose we started a test program that emulates > a virtual USB device. How do we know that the test is over? That depends on what you are testing. > Once all > the hub events are processed we can tell that the device is probably > either connected successfully or failed to connect. But then the > device driver can start it's own thread for example or register a > timer that does more work with the device. Right. For example, usb-storage uses a work-queue routine for SCSI scanning when a new device is probed. And in that routine, the SCSI layer often delegates some of the scanning work to async helper threads, depending on the how the system is configured. > There are probably some > ways to tell what the device driver does at this moment for some > standard USB classes. Yes, I imagine so. But it will be different for each driver. > I guess I'll start with a timeout for now, until I figure out how to > properly collect coverage. That's probably the best thing to do. ... > > In fact, dummy-hcd is capable of creating more than one virtual root > > hub (see the "num" module parameter), so in theory you could run > > multiple tests at the same time. I don't think this capability has > > been tested very much. > > OK, that's nice. I just tried it and it kind of works, but there are two > issues: > > 1. There's a hardcoded limit on the maximum number of virtual UDCs > (#define MAX_NUM_UDC 2). Increasing this parameters seems to work just > fine. > > 2. All the virtual UDCs get the same name "dummy_udc". And since the > process of UDC lookup to bind a gadget driver is based on the UDC name > (see usb_gadget_probe_driver()), we can't bind different gadget > drivers to different UDCs (and therefore connect to different hubs). That can be changed pretty easily. For virtual UDCs beyond the first, we could append "-2", "-3", and so on to the "dummy_udc" name, for example. > >> Basically we need a way to run a test program (that connects a USB > >> device and works with it from userspace for example) and then destroy > >> all the accumulated state before running the next test. > > > > You can do that with dummy-hcd simply by removing the USB gadget > > (closing the gadgetfs files, for example). > > Do I understand correctly that calling usb_gadget_unregister_driver() > should get rid of all of the device state? (That's what gadgetfs seems > to do when you close the file). It should. If it doesn't, there's a bug in the UDC driver or UDC core. > > If you want to be extra > > thorough, unload the gadget driver and dummy-hcd modules after the end > > of the test, and then reload them before starting the next test. That > > should get rid of any important state. > > What state is kept if we don't reload the module? There shouldn't be anything important. Perhaps just some trivial leftovers (like
Re: External USB fuzzing
On Mon, Jun 4, 2018 at 9:37 PM, Alan Stern wrote: > On Mon, 4 Jun 2018, Andrey Konovalov wrote: Hi, Alan! [...] >> The perfect solution would be to have something like /dev/tun for USB, >> where you can write USB packets and the kernel would synchronously >> process them. I'm not sure whether this is possible (highly >> asynchronous USB core subsystem architecture + all communication is >> initiated by the host). > > gadgetfs is a little like that already, except of course that the > processing is not synchronous. Yes, I started with GadgetFS initially, but actually ended up writing my own interface that provides userspace interface to the gadget API instead. Once I figure out what exactly I need from it, it might be a good idea to merge it with GadgetFS as some special mode. Or not, not sure yet. > > Another aspect of this would be fuzzing USB host controller drivers. > In theory you could start doing this now, although you would have to > use a real UDC instead of dummy-hcd. However, this may create more > complications that you want to deal with at the moment. Could you elaborate on this? AFAIU the dummy device contains both a virtual UDC and a virtual HCD that are connected to each other and allow to plug a gadget device into usb core within the kernel. How do you test a driver for an actual host controller? Is there a way to connect a dummy UDC with an actual HCD? > >> Another thing we could do is to teach kcov to collect coverage from >> arbitrary kernel threads. In this case we could collect coverage from >> the thread that processes the USB events. > > That seems like a more general solution, something which could be > applied to other settings too. One difficulty you would face is > telling the tool which threads to work on. This is particularly tricky > when dealing with work queues, since a single work-queue thread can run > code for many different drivers and purposes. > > Perhaps it would be better to collect coverage information based on the > location of the code being executed rather than the thread ID. Yes, that is what I was thinking about. I have a prototype patch for kcov that allows to annotate a block of the kernel code and collect coverage from that block. I think this is the way I'll try to move forward with. > >> However we would still need >> some signal to understand whether the device finished initialization, >> etc. (with the first approach we could tell that some stage of device >> initialization is finished by seeing that hub_event() returned). > > Why initialization particularly? A device goes through a number of > stages in its life cycle. > > I get the impression you would like to have a mechanism whereby you can > send messages from arbitrary places in the kernel to your testing > process. A little like the function tracer or perf, perhaps. Yes, we need to see all the stages, initialization was just an example. The main question is: suppose we started a test program that emulates a virtual USB device. How do we know that the test is over? Once all the hub events are processed we can tell that the device is probably either connected successfully or failed to connect. But then the device driver can start it's own thread for example or register a timer that does more work with the device. There are probably some ways to tell what the device driver does at this moment for some standard USB classes. I guess I'll start with a timeout for now, until I figure out how to properly collect coverage. [...] >> 2. A way to isolate independent test programs. >> >> Right now CONFIG_USB_DUMMY_HCD creates one global hub to which all >> emulated USB devices connect. Would it be possible to create one >> virtual hub per test (or at least one per test process) by say calling >> some ioctl? What changes would that require? > > Are you planning to run multiple test processes concurrently? If you > aren't then this isn't an issue. Yes, I believe that would increase fuzzing speed. > > In fact, dummy-hcd is capable of creating more than one virtual root > hub (see the "num" module parameter), so in theory you could run > multiple tests at the same time. I don't think this capability has > been tested very much. OK, that's nice. I just tried it and it kind of works, but there are two issues: 1. There's a hardcoded limit on the maximum number of virtual UDCs (#define MAX_NUM_UDC 2). Increasing this parameters seems to work just fine. 2. All the virtual UDCs get the same name "dummy_udc". And since the process of UDC lookup to bind a gadget driver is based on the UDC name (see usb_gadget_probe_driver()), we can't bind different gadget drivers to different UDCs (and therefore connect to different hubs). > >> Basically we need a way to run a test program (that connects a USB >> device and works with it from userspace for example) and then destroy >> all the accumulated state before running the next test. > > You can do that with dummy-hcd simply by removing the USB gadget
Re: External USB fuzzing
On Mon, 4 Jun 2018, Andrey Konovalov wrote: > Hi Greg and Alan! > > As you might know I've been working on adding external USB fuzzing > support to syzkaller. Yes. It's an ambitious goal. > At this point a have a prototype, which is able to emulate USB devices > from userspace via a custom written userspace interface for the gadget > subsystem and CONFIG_USB_DUMMY_HCD. The patches and some details are > here [1]. > > This is just a prototype and there are a few things we need to support > USB fuzzing properly. > > 1. A way to collect coverage from USB code. > > Syzkaller uses kcov (CONFIG_KCOV) to collect coverage from the kernel. > The issue is that kcov collects coverage in a process context (from > the kernel task that corresponds to the userspace process that created > the kcov device). But AFAIU USB uses a kernel worker to process USB > hub events. That's right. And other parts of the USB stack do this too, not just the hub driver. Some drivers create their own helper threads, some rely on work queues, some rely on threaded interrupt handlers, and so on. Not to mention non-process stuff, like timer or softirq callbacks. > The approach I tried to deal with this is to change hub_event() to put > events into a global queue and wait instead of processing them right > away. Then I added an ioctl to the hub device that takes an event from > the queue and processes it the same way hub_event() used to. This > results in USB events being processed in a process context. The patch > is here [2]. > > Is there a less hacky way to make processing USB events synchronously > (by an ioctl call from userspace for example), if that's feasible at > all? Sounds exactly like what you have already done for the hub driver. It might be okay in that setting, and maybe in a few other drivers, but it isn't a general-purpose solution. In general, I don't think that kind of approach will work for all drivers. There are too many different asynchronous execution mechanisms in use. Not to mention execution pathways that originate completely outside the USB stack (Power Management callbacks). > The perfect solution would be to have something like /dev/tun for USB, > where you can write USB packets and the kernel would synchronously > process them. I'm not sure whether this is possible (highly > asynchronous USB core subsystem architecture + all communication is > initiated by the host). gadgetfs is a little like that already, except of course that the processing is not synchronous. Another aspect of this would be fuzzing USB host controller drivers. In theory you could start doing this now, although you would have to use a real UDC instead of dummy-hcd. However, this may create more complications that you want to deal with at the moment. > Another thing we could do is to teach kcov to collect coverage from > arbitrary kernel threads. In this case we could collect coverage from > the thread that processes the USB events. That seems like a more general solution, something which could be applied to other settings too. One difficulty you would face is telling the tool which threads to work on. This is particularly tricky when dealing with work queues, since a single work-queue thread can run code for many different drivers and purposes. Perhaps it would be better to collect coverage information based on the location of the code being executed rather than the thread ID. > However we would still need > some signal to understand whether the device finished initialization, > etc. (with the first approach we could tell that some stage of device > initialization is finished by seeing that hub_event() returned). Why initialization particularly? A device goes through a number of stages in its life cycle. I get the impression you would like to have a mechanism whereby you can send messages from arbitrary places in the kernel to your testing process. A little like the function tracer or perf, perhaps. > There's another issue with getting coverage after USB device has been > initialized. For example some device drivers create new threads that > communicate with that device. I don't see an easy way to collect > coverage from those threads. Right. This is the problem I mentioned above. > 2. A way to isolate independent test programs. > > Right now CONFIG_USB_DUMMY_HCD creates one global hub to which all > emulated USB devices connect. Would it be possible to create one > virtual hub per test (or at least one per test process) by say calling > some ioctl? What changes would that require? Are you planning to run multiple test processes concurrently? If you aren't then this isn't an issue. In fact, dummy-hcd is capable of creating more than one virtual root hub (see the "num" module parameter), so in theory you could run multiple tests a
External USB fuzzing
Hi Greg and Alan! As you might know I've been working on adding external USB fuzzing support to syzkaller. At this point a have a prototype, which is able to emulate USB devices from userspace via a custom written userspace interface for the gadget subsystem and CONFIG_USB_DUMMY_HCD. The patches and some details are here [1]. This is just a prototype and there are a few things we need to support USB fuzzing properly. 1. A way to collect coverage from USB code. Syzkaller uses kcov (CONFIG_KCOV) to collect coverage from the kernel. The issue is that kcov collects coverage in a process context (from the kernel task that corresponds to the userspace process that created the kcov device). But AFAIU USB uses a kernel worker to process USB hub events. The approach I tried to deal with this is to change hub_event() to put events into a global queue and wait instead of processing them right away. Then I added an ioctl to the hub device that takes an event from the queue and processes it the same way hub_event() used to. This results in USB events being processed in a process context. The patch is here [2]. Is there a less hacky way to make processing USB events synchronously (by an ioctl call from userspace for example), if that's feasible at all? The perfect solution would be to have something like /dev/tun for USB, where you can write USB packets and the kernel would synchronously process them. I'm not sure whether this is possible (highly asynchronous USB core subsystem architecture + all communication is initiated by the host). Another thing we could do is to teach kcov to collect coverage from arbitrary kernel threads. In this case we could collect coverage from the thread that processes the USB events. However we would still need some signal to understand whether the device finished initialization, etc. (with the first approach we could tell that some stage of device initialization is finished by seeing that hub_event() returned). There's another issue with getting coverage after USB device has been initialized. For example some device drivers create new threads that communicate with that device. I don't see an easy way to collect coverage from those threads. 2. A way to isolate independent test programs. Right now CONFIG_USB_DUMMY_HCD creates one global hub to which all emulated USB devices connect. Would it be possible to create one virtual hub per test (or at least one per test process) by say calling some ioctl? What changes would that require? Basically we need a way to run a test program (that connects a USB device and works with it from userspace for example) and then destroy all the accumulated state before running the next test. Thanks! [1] https://github.com/google/syzkaller/blob/usb-fuzzer/docs/linux/external_fuzzing_usb.md [2] https://github.com/google/syzkaller/blob/usb-fuzzer/tools/usb/0001-usb-fuzzer-add-hub-hijacking-ioctl.patch -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
