Re: USB driver resets
On 19/12/2018 15:07, Mathias Nyman wrote: On 19.12.2018 12:18, Richard van der Hoff wrote: On 13/12/2018 10:29, Richard van der Hoff wrote: Thanks for looking at it. I've attached a log from a 4.19.8 kernel showing the problem with the dynamic debug flags you suggested. The first instance of the problem seems to begin with: Dec 13 09:32:56 fred kernel: [ 479.158038] xhci_hcd :39:00.0: Port Status Change Event for port 1 Dec 13 09:32:56 fred kernel: [ 479.158045] xhci_hcd :39:00.0: handle_port_status: starting port polling. Dec 13 09:32:56 fred kernel: [ 479.158062] hub 3-0:1.0: state 7 ports 2 chg evt 0002 Dec 13 09:32:56 fred kernel: [ 479.158070] xhci_hcd :39:00.0: get port status, actual port 0 status = 0x202a0 Dec 13 09:32:56 fred kernel: [ 479.158071] xhci_hcd :39:00.0: Get port status returned 0x10100 Dec 13 09:32:56 fred kernel: [ 479.158083] xhci_hcd :39:00.0: clear port connect change, actual port 0 status = 0x2a0 Dec 13 09:32:56 fred kernel: [ 479.158087] usb usb3-port1: status 0100, change 0001, 12 Mb/s Dec 13 09:32:56 fred kernel: [ 479.158095] usb 3-1: USB disconnect, device number 2 Dec 13 09:32:56 fred kernel: [ 479.158097] usb 3-1.1: USB disconnect, device number 3 Dec 13 09:32:56 fred kernel: [ 479.158098] usb 3-1.1.3: USB disconnect, device number 8 Dec 13 09:32:56 fred kernel: [ 479.158100] usb 3-1.1.3: unregistering device Mika, Mathias: sorry to pester. I've been trying to decode this with the help of the source. Any clues? My impression is that "get port status, actual port 0 status = 0x202a0" means that the xhci controller is reporting a zero value for the port status and control register - ie, that the dock is disconnected. If that's the case, it presumably points fairly firmly to a hardware issue of some kind. It would be really helpful to get your thoughts on this. Sorry about the late reply, you are right, xhci reports that something happened on that port, and when reading PORSC register it shows a device was disconnected. When the last USB device connected to a Alpine Ridge xHC controller the whole controller is PCI hotplug removed from the PCI bus. Details: * Dec 13 09:32:56 fred kernel: [ 479.158038] xhci_hcd :39:00.0: Port Status Change Event for port 1 means xhci controller sent an interrupt because it something changes on a port * Dec 13 09:32:56 fred kernel: [ 479.158070] xhci_hcd :39:00.0: get port status, actual port 0 status = 0x202a0 Port Status: 0x202a0 Disconnected Disabled Link: Rx Detect Powered Unknown port speed Connect Status Change Connection changed to disconnect, this is what a port status should look like when a SS USB 3 cable is unplugged. The xHCI controller at :39:00.0 creates two USB buses, a High Speed (HS) bus named usb3, and a SuperSpeed (SS) bus named usb4. A SS hub will have both a HS and SS hub built into it. The internal roothub in xHCI is shown as a HS hub at the usb3 bus named named usb3-0, and a SS hub at usb4-0. The dock device is seen as a HS hub at usb3-1, and SS hub at usb4-1. All other devices in your dock are behind these hubs. We see a sudden disconnect of the HS parts of the external hub, most likely the SS hub part would disconnect soon, but the Alpine ridge controller is very fast in hotplug removing the whole xHCI contoller from PCI bus once no usb devices are connected, so next we see PCI code calling the remove callback of the xHC PCI device: * Dec 13 09:32:56 fred kernel: [ 479.260618] xhci_hcd :39:00.0: remove, state 1 And later xHC controller is removed from PCI bus: *Dec 13 09:32:57 fred kernel: [ 479.609866] pci_bus :39: busn_res: [bus 39] is released Until apparently a connected usb device is discovered afain, xHC controller is added to pci bus, xhci driver is loaded and so on. So main cause here is a sudden disconnect of the dock. Right, thanks very much for explaining all that. I'll take it up with Plugable support. Richard
Re: USB driver resets
On 19.12.2018 12:18, Richard van der Hoff wrote: On 13/12/2018 10:29, Richard van der Hoff wrote: Thanks for looking at it. I've attached a log from a 4.19.8 kernel showing the problem with the dynamic debug flags you suggested. The first instance of the problem seems to begin with: Dec 13 09:32:56 fred kernel: [ 479.158038] xhci_hcd :39:00.0: Port Status Change Event for port 1 Dec 13 09:32:56 fred kernel: [ 479.158045] xhci_hcd :39:00.0: handle_port_status: starting port polling. Dec 13 09:32:56 fred kernel: [ 479.158062] hub 3-0:1.0: state 7 ports 2 chg evt 0002 Dec 13 09:32:56 fred kernel: [ 479.158070] xhci_hcd :39:00.0: get port status, actual port 0 status = 0x202a0 Dec 13 09:32:56 fred kernel: [ 479.158071] xhci_hcd :39:00.0: Get port status returned 0x10100 Dec 13 09:32:56 fred kernel: [ 479.158083] xhci_hcd :39:00.0: clear port connect change, actual port 0 status = 0x2a0 Dec 13 09:32:56 fred kernel: [ 479.158087] usb usb3-port1: status 0100, change 0001, 12 Mb/s Dec 13 09:32:56 fred kernel: [ 479.158095] usb 3-1: USB disconnect, device number 2 Dec 13 09:32:56 fred kernel: [ 479.158097] usb 3-1.1: USB disconnect, device number 3 Dec 13 09:32:56 fred kernel: [ 479.158098] usb 3-1.1.3: USB disconnect, device number 8 Dec 13 09:32:56 fred kernel: [ 479.158100] usb 3-1.1.3: unregistering device Mika, Mathias: sorry to pester. I've been trying to decode this with the help of the source. Any clues? My impression is that "get port status, actual port 0 status = 0x202a0" means that the xhci controller is reporting a zero value for the port status and control register - ie, that the dock is disconnected. If that's the case, it presumably points fairly firmly to a hardware issue of some kind. It would be really helpful to get your thoughts on this. Sorry about the late reply, you are right, xhci reports that something happened on that port, and when reading PORSC register it shows a device was disconnected. When the last USB device connected to a Alpine Ridge xHC controller the whole controller is PCI hotplug removed from the PCI bus. Details: * Dec 13 09:32:56 fred kernel: [ 479.158038] xhci_hcd :39:00.0: Port Status Change Event for port 1 means xhci controller sent an interrupt because it something changes on a port * Dec 13 09:32:56 fred kernel: [ 479.158070] xhci_hcd :39:00.0: get port status, actual port 0 status = 0x202a0 Port Status: 0x202a0 Disconnected Disabled Link: Rx Detect Powered Unknown port speed Connect Status Change Connection changed to disconnect, this is what a port status should look like when a SS USB 3 cable is unplugged. The xHCI controller at :39:00.0 creates two USB buses, a High Speed (HS) bus named usb3, and a SuperSpeed (SS) bus named usb4. A SS hub will have both a HS and SS hub built into it. The internal roothub in xHCI is shown as a HS hub at the usb3 bus named named usb3-0, and a SS hub at usb4-0. The dock device is seen as a HS hub at usb3-1, and SS hub at usb4-1. All other devices in your dock are behind these hubs. We see a sudden disconnect of the HS parts of the external hub, most likely the SS hub part would disconnect soon, but the Alpine ridge controller is very fast in hotplug removing the whole xHCI contoller from PCI bus once no usb devices are connected, so next we see PCI code calling the remove callback of the xHC PCI device: * Dec 13 09:32:56 fred kernel: [ 479.260618] xhci_hcd :39:00.0: remove, state 1 And later xHC controller is removed from PCI bus: *Dec 13 09:32:57 fred kernel: [ 479.609866] pci_bus :39: busn_res: [bus 39] is released Until apparently a connected usb device is discovered afain, xHC controller is added to pci bus, xhci driver is loaded and so on. So main cause here is a sudden disconnect of the dock. -Mathias
Re: USB driver resets
On 13/12/2018 10:29, Richard van der Hoff wrote: Thanks for looking at it. I've attached a log from a 4.19.8 kernel showing the problem with the dynamic debug flags you suggested. The first instance of the problem seems to begin with: Dec 13 09:32:56 fred kernel: [ 479.158038] xhci_hcd :39:00.0: Port Status Change Event for port 1 Dec 13 09:32:56 fred kernel: [ 479.158045] xhci_hcd :39:00.0: handle_port_status: starting port polling. Dec 13 09:32:56 fred kernel: [ 479.158062] hub 3-0:1.0: state 7 ports 2 chg evt 0002 Dec 13 09:32:56 fred kernel: [ 479.158070] xhci_hcd :39:00.0: get port status, actual port 0 status = 0x202a0 Dec 13 09:32:56 fred kernel: [ 479.158071] xhci_hcd :39:00.0: Get port status returned 0x10100 Dec 13 09:32:56 fred kernel: [ 479.158083] xhci_hcd :39:00.0: clear port connect change, actual port 0 status = 0x2a0 Dec 13 09:32:56 fred kernel: [ 479.158087] usb usb3-port1: status 0100, change 0001, 12 Mb/s Dec 13 09:32:56 fred kernel: [ 479.158095] usb 3-1: USB disconnect, device number 2 Dec 13 09:32:56 fred kernel: [ 479.158097] usb 3-1.1: USB disconnect, device number 3 Dec 13 09:32:56 fred kernel: [ 479.158098] usb 3-1.1.3: USB disconnect, device number 8 Dec 13 09:32:56 fred kernel: [ 479.158100] usb 3-1.1.3: unregistering device Mika, Mathias: sorry to pester. I've been trying to decode this with the help of the source. Any clues? My impression is that "get port status, actual port 0 status = 0x202a0" means that the xhci controller is reporting a zero value for the port status and control register - ie, that the dock is disconnected. If that's the case, it presumably points fairly firmly to a hardware issue of some kind. It would be really helpful to get your thoughts on this.
Re: USB driver resets
On Mon, Dec 10, 2018 at 03:02:26PM +, Richard van der Hoff wrote: > On 10/12/2018 14:53, Mika Westerberg wrote: > > On Mon, Dec 10, 2018 at 01:33:45PM +, Richard van der Hoff wrote: > > > > > > On 10/12/2018 13:30, Mathias Nyman wrote: > > > > Do you have the beginning part of the dmesg? It could show why the USB > > > > devices all get disconnected. > > > > The whole xhci usb host will be hotplug removed from the PCI bus when > > > > last device is disconnected, this > > > > is roughly where your current dmesg starts. > > > > > > There's nothing in the dmesg for many minutes before what I sent. > > > > It still might reveal something useful, like PCI resource allocation > > etc. So it is always good idea to provide a full dmesg. > > > My bad. Here you go. Thanks for the log. Unfortunately I can't immediately spot anything that could explain this. > [ 541.184315] IPv6: ADDRCONF(NETDEV_UP): enx00249b194c4f: link is not ready > [ 1943.745174] uvcvideo: Failed to query (GET_CUR) UVC control 11 on unit 2: > -110 (exp. 1). > [ 1987.411132] usb 3-1: USB disconnect, device number 2 > [ 1987.411135] usb 3-1.1: USB disconnect, device number 3 > [ 1987.411137] usb 3-1.1.3: USB disconnect, device number 5 > [ 1987.411553] usb 3-1.1.3: cannot submit urb (err = -19) > [ 1987.412833] usb 3-1.1.3: cannot submit urb (err = -19) > [ 1987.470866] xhci_hcd :39:00.0: remove, state 1 To me this looks pretty much like the USB link goes down for unknown reasons and that triggers the ACPI PCI hot-remove followed by hot-add again (these systems the xHCI controller is removed if there is nothing connected). Maybe Mathias and other USB developers are able to figure out more.
Re: USB driver resets
On Mon, Dec 10, 2018 at 01:33:45PM +, Richard van der Hoff wrote: > > On 10/12/2018 13:30, Mathias Nyman wrote: > > Do you have the beginning part of the dmesg? It could show why the USB > > devices all get disconnected. > > The whole xhci usb host will be hotplug removed from the PCI bus when > > last device is disconnected, this > > is roughly where your current dmesg starts. > > There's nothing in the dmesg for many minutes before what I sent. It still might reveal something useful, like PCI resource allocation etc. So it is always good idea to provide a full dmesg.
Re: USB driver resets
On 10/12/2018 13:30, Mathias Nyman wrote: Do you have the beginning part of the dmesg? It could show why the USB devices all get disconnected. The whole xhci usb host will be hotplug removed from the PCI bus when last device is disconnected, this is roughly where your current dmesg starts. There's nothing in the dmesg for many minutes before what I sent. Also adding xhci, and usb core dynamic debug could help: mount -t debugfs none /sys/kernel/debug echo -n 'module xhci_hcd =p' > /sys/kernel/debug/dynamic_debug/control echo -n 'module usbcore =p' >/sys/kernel/debug/dynamic_debug/control Noted. Will give them a go and try to reproduce.
Re: USB driver resets
On 10.12.2018 14:24, Richard van der Hoff wrote: On 05/12/2018 10:08, Richard van der Hoff wrote: On 05/12/2018 09:44, Mika Westerberg wrote: On Tue, Dec 04, 2018 at 04:02:56PM +, Richard van der Hoff wrote: Sorry. The system is a Dell XPS13 9350, with an Intel DSL6340 Thunderbolt 3 bridge. The dock is a Plugable UD-CA1 [2]. Also if you have dual boot or similar, do you experience similar issue in Windows side? I do have dual-boot, though I very rarely boot into Windows. Since the issue is somewhat intermittent (for some reason it seems to happen particularly when I'm in a video conference...) I wouldn't have a huge amount of confidence of being able to reproduce it under Windows. Thanks again. [1] https://bugzilla.kernel.org/show_bug.cgi?id=201853 [2] https://plugable.com/products/ud-ca1/ This seems to be normal USB-C dock (not TBT). Yes, that's my understanding too. In the logs it looks like at some point the USB link just goes down which makes the host side to hot-remove xHCI and the intermediate PCIe ports. The UD-CA1 dock site you linked above mentions connection issues with some systems so I'm suspecting it is an issue with the dock itself not with the kernel. It would be good if you could try to reproduce on Windows side as well, basically doing the same things you do in Linux (assuming it is possible) and see if it triggers. OK, will give it a go. It'll probably take me a few days. So I've not managed to reproduce the problem with Windows, despite doing the same things that normally trigger it under Linux. That of course doesn't tell us anything terribly conclusive in itself. Any suggestions for next steps? I'd be happy to try taking it up with Plugable support if you think there's evidence that the problem is with the dock. Do you have the beginning part of the dmesg? It could show why the USB devices all get disconnected. The whole xhci usb host will be hotplug removed from the PCI bus when last device is disconnected, this is roughly where your current dmesg starts. Also adding xhci, and usb core dynamic debug could help: mount -t debugfs none /sys/kernel/debug echo -n 'module xhci_hcd =p' > /sys/kernel/debug/dynamic_debug/control echo -n 'module usbcore =p' >/sys/kernel/debug/dynamic_debug/control -Mathias
Re: USB driver resets
On 05/12/2018 10:08, Richard van der Hoff wrote: On 05/12/2018 09:44, Mika Westerberg wrote: On Tue, Dec 04, 2018 at 04:02:56PM +, Richard van der Hoff wrote: Sorry. The system is a Dell XPS13 9350, with an Intel DSL6340 Thunderbolt 3 bridge. The dock is a Plugable UD-CA1 [2]. Also if you have dual boot or similar, do you experience similar issue in Windows side? I do have dual-boot, though I very rarely boot into Windows. Since the issue is somewhat intermittent (for some reason it seems to happen particularly when I'm in a video conference...) I wouldn't have a huge amount of confidence of being able to reproduce it under Windows. Thanks again. [1] https://bugzilla.kernel.org/show_bug.cgi?id=201853 [2] https://plugable.com/products/ud-ca1/ This seems to be normal USB-C dock (not TBT). Yes, that's my understanding too. In the logs it looks like at some point the USB link just goes down which makes the host side to hot-remove xHCI and the intermediate PCIe ports. The UD-CA1 dock site you linked above mentions connection issues with some systems so I'm suspecting it is an issue with the dock itself not with the kernel. It would be good if you could try to reproduce on Windows side as well, basically doing the same things you do in Linux (assuming it is possible) and see if it triggers. OK, will give it a go. It'll probably take me a few days. So I've not managed to reproduce the problem with Windows, despite doing the same things that normally trigger it under Linux. That of course doesn't tell us anything terribly conclusive in itself. Any suggestions for next steps? I'd be happy to try taking it up with Plugable support if you think there's evidence that the problem is with the dock.
Re: USB driver resets
On 05/12/2018 09:44, Mika Westerberg wrote: On Tue, Dec 04, 2018 at 04:02:56PM +, Richard van der Hoff wrote: Sorry. The system is a Dell XPS13 9350, with an Intel DSL6340 Thunderbolt 3 bridge. The dock is a Plugable UD-CA1 [2]. Also if you have dual boot or similar, do you experience similar issue in Windows side? I do have dual-boot, though I very rarely boot into Windows. Since the issue is somewhat intermittent (for some reason it seems to happen particularly when I'm in a video conference...) I wouldn't have a huge amount of confidence of being able to reproduce it under Windows. Thanks again. [1] https://bugzilla.kernel.org/show_bug.cgi?id=201853 [2] https://plugable.com/products/ud-ca1/ This seems to be normal USB-C dock (not TBT). Yes, that's my understanding too. In the logs it looks like at some point the USB link just goes down which makes the host side to hot-remove xHCI and the intermediate PCIe ports. The UD-CA1 dock site you linked above mentions connection issues with some systems so I'm suspecting it is an issue with the dock itself not with the kernel. It would be good if you could try to reproduce on Windows side as well, basically doing the same things you do in Linux (assuming it is possible) and see if it triggers. OK, will give it a go. It'll probably take me a few days. Thanks for your help.
Re: USB driver resets
Hi, On Tue, Dec 04, 2018 at 04:02:56PM +, Richard van der Hoff wrote: > Mathias, Mika, thanks very much for your help. > > On 04/12/2018 15:09, Mika Westerberg wrote: > > On Tue, Dec 04, 2018 at 04:45:33PM +0200, Mathias Nyman wrote: > > > > On 19/11/2018 15:27, Richard van der Hoff wrote: > > > > > I have a USB-3.1 dock, which I use for video (via displayport alt > > > > > mode) and power delivery, as well as keyboard, mouse, etc. > > > > > > > > > > From time to time, all connectivity with the dock drops out for a few > > > > > seconds, before resetting itself. I'm trying to pin down if this is a > > > > > problem with the usb drivers, pci drivers, laptop hardware, or the > > > > > dock. It looks to me from the dmesg output like it could be a problem > > > > > at the PCI layer, but I am only speculating really. > > > > > > > > > > Attached are the dmesg output during the problem, lsusb -v, and lspci > > > > > -v. They are all recorded with a 4.15.0 kernel; I've seen exactly the > > > > > same symptoms with a number of kernels between 4.8 and 4.15. > > > > > > > > > > > Mika (cc) would be the right person to contact about these kind of > > > thunderbolt related issues. > > > I'm sure he would appreciate logs with a 4.19 - 4.20-rcx kernel. > > I've attached the kernel log output from a 4.19 kernel. I've actually had > some other problems with a 4.19 kernel [1] so have gone back to the 4.15 > kernel for now, but please do let me know if there is anything specific I > should try. > > I've also re-attached the lsusb -v and lspci -v output (from a 4.15 kernel) > for your reference. > > > Right, those would help and also some knowledge about which system and > > dock we are dealing with :) > > Sorry. The system is a Dell XPS13 9350, with an Intel DSL6340 Thunderbolt 3 > bridge. > > The dock is a Plugable UD-CA1 [2]. > > > Also if you have dual boot or similar, do you experience similar issue > > in Windows side? > > I do have dual-boot, though I very rarely boot into Windows. Since the issue > is somewhat intermittent (for some reason it seems to happen particularly > when I'm in a video conference...) I wouldn't have a huge amount of > confidence of being able to reproduce it under Windows. > > Thanks again. > > > [1] https://bugzilla.kernel.org/show_bug.cgi?id=201853 > [2] https://plugable.com/products/ud-ca1/ This seems to be normal USB-C dock (not TBT). In the logs it looks like at some point the USB link just goes down which makes the host side to hot-remove xHCI and the intermediate PCIe ports. The UD-CA1 dock site you linked above mentions connection issues with some systems so I'm suspecting it is an issue with the dock itself not with the kernel. It would be good if you could try to reproduce on Windows side as well, basically doing the same things you do in Linux (assuming it is possible) and see if it triggers.
Re: USB driver resets
On Tue, Dec 04, 2018 at 04:45:33PM +0200, Mathias Nyman wrote: > On 03.12.2018 12:36, Richard van der Hoff wrote: > > Does anybody have any suggestions as to how I could set about debugging > > this? I'm seeing the same symptoms on a 4.19 kernel. > > > > > > On 19/11/2018 15:27, Richard van der Hoff wrote: > > > I have a USB-3.1 dock, which I use for video (via displayport alt mode) > > > and power delivery, as well as keyboard, mouse, etc. > > > > > > From time to time, all connectivity with the dock drops out for a few > > > seconds, before resetting itself. I'm trying to pin down if this is a > > > problem with the usb drivers, pci drivers, laptop hardware, or the dock. > > > It looks to me from the dmesg output like it could be a problem at the > > > PCI layer, but I am only speculating really. > > > > > > Attached are the dmesg output during the problem, lsusb -v, and lspci -v. > > > They are all recorded with a 4.15.0 kernel; I've seen exactly the same > > > symptoms with a number of kernels between 4.8 and 4.15. > > > > > > Advice appreciated! > > > > > Mika (cc) would be the right person to contact about these kind of > thunderbolt related issues. > I'm sure he would appreciate logs with a 4.19 - 4.20-rcx kernel. Right, those would help and also some knowledge about which system and dock we are dealing with :) Also if you have dual boot or similar, do you experience similar issue in Windows side?
Re: USB driver resets
On 03.12.2018 12:36, Richard van der Hoff wrote: Does anybody have any suggestions as to how I could set about debugging this? I'm seeing the same symptoms on a 4.19 kernel. On 19/11/2018 15:27, Richard van der Hoff wrote: I have a USB-3.1 dock, which I use for video (via displayport alt mode) and power delivery, as well as keyboard, mouse, etc. From time to time, all connectivity with the dock drops out for a few seconds, before resetting itself. I'm trying to pin down if this is a problem with the usb drivers, pci drivers, laptop hardware, or the dock. It looks to me from the dmesg output like it could be a problem at the PCI layer, but I am only speculating really. Attached are the dmesg output during the problem, lsusb -v, and lspci -v. They are all recorded with a 4.15.0 kernel; I've seen exactly the same symptoms with a number of kernels between 4.8 and 4.15. Advice appreciated! Mika (cc) would be the right person to contact about these kind of thunderbolt related issues. I'm sure he would appreciate logs with a 4.19 - 4.20-rcx kernel. -Mathias
Re: USB driver resets
Does anybody have any suggestions as to how I could set about debugging this? I'm seeing the same symptoms on a 4.19 kernel. On 19/11/2018 15:27, Richard van der Hoff wrote: I have a USB-3.1 dock, which I use for video (via displayport alt mode) and power delivery, as well as keyboard, mouse, etc. From time to time, all connectivity with the dock drops out for a few seconds, before resetting itself. I'm trying to pin down if this is a problem with the usb drivers, pci drivers, laptop hardware, or the dock. It looks to me from the dmesg output like it could be a problem at the PCI layer, but I am only speculating really. Attached are the dmesg output during the problem, lsusb -v, and lspci -v. They are all recorded with a 4.15.0 kernel; I've seen exactly the same symptoms with a number of kernels between 4.8 and 4.15. Advice appreciated!
