Re: [PATCH v3] PCI: hosts: mark pcie/pci (msi) irq cascade handler as IRQF_NO_THREAD

[Bjorn Helgaas]

Hi Grygorii,

On Thu, Dec 10, 2015 at 09:18:20PM +0200, Grygorii Strashko wrote:
> On -RT and if kernel is booting with "threadirqs" cmd line parameter
> pcie/pci (msi) irq cascade handlers (like dra7xx_pcie_msi_irq_handler())
> will be forced threaded and, as result, will generate warnings like:
> 
> WARNING: CPU: 1 PID: 82 at kernel/irq/handle.c:150 
> handle_irq_event_percpu+0x14c/0x174()
> irq 460 handler irq_default_primary_handler+0x0/0x14 enabled interrupts
> Backtrace:
>  (warn_slowpath_common) from [] (warn_slowpath_fmt+0x38/0x40)
>  (warn_slowpath_fmt) from [] (handle_irq_event_percpu+0x14c/0x174)
>  (handle_irq_event_percpu) from [] (handle_irq_event+0x84/0xb8)
>  (handle_irq_event) from [] (handle_simple_irq+0x90/0x118)
>  (handle_simple_irq) from [] (generic_handle_irq+0x30/0x44)
>  (generic_handle_irq) from [] 
> (dra7xx_pcie_msi_irq_handler+0x7c/0x8c)
>  (dra7xx_pcie_msi_irq_handler) from [] 
> (irq_forced_thread_fn+0x28/0x5c)
>  (irq_forced_thread_fn) from [] (irq_thread+0x128/0x204)
> 
> This happens because all of them invoke generic_handle_irq() from the
> requsted handler. generic_handle_irq grabs raw_locks and this needs to
> run in raw-irq context.
> 
> This issue was originally reproduced on TI dra7-evem, but, as was
> identified during dicussion [1], other PCI(e) hosts can also suffer
> from this issue. So let's fix all them at once and mark pcie/pci (msi)
> irq cascade handlers IRQF_NO_THREAD explicitly.
> 
> [1] https://lkml.org/lkml/2015/11/20/356
> 
> Cc: Kishon Vijay Abraham I 
> Cc: Bjorn Helgaas 
> Cc: Jingoo Han 
> Cc: Kukjin Kim 
> Cc: Krzysztof Kozlowski 
> Cc: Richard Zhu 
> Cc: Lucas Stach 
> Cc: Thierry Reding 
> Cc: Stephen Warren 
> Cc: Alexandre Courbot 
> Cc: Simon Horman 
> Cc: Pratyush Anand 
> Cc: Michal Simek 
> Cc: "Sören Brinkmann" 
> Cc: Sebastian Andrzej Siewior 
> Signed-off-by: Grygorii Strashko 
> ---
> Changes in v3:
>  - change applied to all affected pci(e) host drivers in drivers/pci/hosts.
>After some invsetigation I've decided to not touch arch code - it is not 
> easy
>to identify all places which need to be fixed. 
>if it's still required - i can send separate patches for 
>arch/mips/pci/msi-octeon.c and arch/sparc/kernel/pci_msi.c.
> Links
> v2: https://lkml.org/lkml/2015/11/20/356
> v1: https://lkml.org/lkml/2015/11/5/593
> ref: https://lkml.org/lkml/2015/11/3/660
> 
>  drivers/pci/host/pci-dra7xx.c | 13 -
>  drivers/pci/host/pci-exynos.c |  3 ++-
>  drivers/pci/host/pci-imx6.c   |  3 ++-
>  drivers/pci/host/pci-tegra.c  |  2 +-
>  drivers/pci/host/pcie-rcar.c  |  6 --
>  drivers/pci/host/pcie-spear13xx.c |  3 ++-
>  drivers/pci/host/pcie-xilinx.c|  3 ++-
>  7 files changed, 25 insertions(+), 8 deletions(-)

I applied this to pci/host for v4.5, thanks.  I added a stable tag.
I haven't seen any acks from the host driver guys, but I will still add
them if I see any in the next few days.

Bjorn

> diff --git a/drivers/pci/host/pci-dra7xx.c b/drivers/pci/host/pci-dra7xx.c
> index 8c36880..0415192 100644
> --- a/drivers/pci/host/pci-dra7xx.c
> +++ b/drivers/pci/host/pci-dra7xx.c
> @@ -301,8 +301,19 @@ static int __init dra7xx_add_pcie_port(struct 
> dra7xx_pcie *dra7xx,
>   return -EINVAL;
>   }
>  
> + /*
> +  * Mark dra7xx_pcie_msi IRQ as IRQF_NO_THREAD
> +  * On -RT and if kernel is booting with "threadirqs" cmd line parameter
> +  * the dra7xx_pcie_msi_irq_handler() will be forced threaded but,
> +  * in the same time, it's IRQ dispatcher and calls generic_handle_irq(),
> +  * which, in turn, will be resolved to handle_simple_irq() call.
> +  * The handle_simple_irq() expected to be called with IRQ disabled, as
> +  * result kernle will display warning:
> +  * "irq XXX handler YYY+0x0/0x14 enabled interrupts".
> +  */
>   ret = devm_request_irq(>dev, pp->irq,
> -dra7xx_pcie_msi_irq_handler, IRQF_SHARED,
> +dra7xx_pcie_msi_irq_handler,
> +IRQF_SHARED | IRQF_NO_THREAD,
>  "dra7-pcie-msi", pp);
>   if (ret) {
>   dev_err(>dev, "failed to request irq\n");
> diff --git a/drivers/pci/host/pci-exynos.c b/drivers/pci/host/pci-exynos.c
> index 01095e1..d997d22 100644
> --- a/drivers/pci/host/pci-exynos.c
> +++ b/drivers/pci/host/pci-exynos.c
> @@ -522,7 +522,8 @@ static int __init exynos_add_pcie_port(struct pcie_port 
> *pp,
>  
>   ret = devm_request_irq(>dev, pp->msi_irq,
>   exynos_pcie_msi_irq_handler,

[PATCH v3] PCI: hosts: mark pcie/pci (msi) irq cascade handler as IRQF_NO_THREAD

[Grygorii Strashko]

On -RT and if kernel is booting with "threadirqs" cmd line parameter
pcie/pci (msi) irq cascade handlers (like dra7xx_pcie_msi_irq_handler())
will be forced threaded and, as result, will generate warnings like:

WARNING: CPU: 1 PID: 82 at kernel/irq/handle.c:150 
handle_irq_event_percpu+0x14c/0x174()
irq 460 handler irq_default_primary_handler+0x0/0x14 enabled interrupts
Backtrace:
 (warn_slowpath_common) from [] (warn_slowpath_fmt+0x38/0x40)
 (warn_slowpath_fmt) from [] (handle_irq_event_percpu+0x14c/0x174)
 (handle_irq_event_percpu) from [] (handle_irq_event+0x84/0xb8)
 (handle_irq_event) from [] (handle_simple_irq+0x90/0x118)
 (handle_simple_irq) from [] (generic_handle_irq+0x30/0x44)
 (generic_handle_irq) from [] (dra7xx_pcie_msi_irq_handler+0x7c/0x8c)
 (dra7xx_pcie_msi_irq_handler) from [] 
(irq_forced_thread_fn+0x28/0x5c)
 (irq_forced_thread_fn) from [] (irq_thread+0x128/0x204)

This happens because all of them invoke generic_handle_irq() from the
requsted handler. generic_handle_irq grabs raw_locks and this needs to
run in raw-irq context.

This issue was originally reproduced on TI dra7-evem, but, as was
identified during dicussion [1], other PCI(e) hosts can also suffer
from this issue. So let's fix all them at once and mark pcie/pci (msi)
irq cascade handlers IRQF_NO_THREAD explicitly.

[1] https://lkml.org/lkml/2015/11/20/356

Cc: Kishon Vijay Abraham I 
Cc: Bjorn Helgaas 
Cc: Jingoo Han 
Cc: Kukjin Kim 
Cc: Krzysztof Kozlowski 
Cc: Richard Zhu 
Cc: Lucas Stach 
Cc: Thierry Reding 
Cc: Stephen Warren 
Cc: Alexandre Courbot 
Cc: Simon Horman 
Cc: Pratyush Anand 
Cc: Michal Simek 
Cc: "Sören Brinkmann" 
Cc: Sebastian Andrzej Siewior 
Signed-off-by: Grygorii Strashko 
---
Changes in v3:
 - change applied to all affected pci(e) host drivers in drivers/pci/hosts.
   After some invsetigation I've decided to not touch arch code - it is not easy
   to identify all places which need to be fixed. 
   if it's still required - i can send separate patches for 
   arch/mips/pci/msi-octeon.c and arch/sparc/kernel/pci_msi.c.
Links
v2: https://lkml.org/lkml/2015/11/20/356
v1: https://lkml.org/lkml/2015/11/5/593
ref: https://lkml.org/lkml/2015/11/3/660

 drivers/pci/host/pci-dra7xx.c | 13 -
 drivers/pci/host/pci-exynos.c |  3 ++-
 drivers/pci/host/pci-imx6.c   |  3 ++-
 drivers/pci/host/pci-tegra.c  |  2 +-
 drivers/pci/host/pcie-rcar.c  |  6 --
 drivers/pci/host/pcie-spear13xx.c |  3 ++-
 drivers/pci/host/pcie-xilinx.c|  3 ++-
 7 files changed, 25 insertions(+), 8 deletions(-)

diff --git a/drivers/pci/host/pci-dra7xx.c b/drivers/pci/host/pci-dra7xx.c
index 8c36880..0415192 100644
--- a/drivers/pci/host/pci-dra7xx.c
+++ b/drivers/pci/host/pci-dra7xx.c
@@ -301,8 +301,19 @@ static int __init dra7xx_add_pcie_port(struct dra7xx_pcie 
*dra7xx,
return -EINVAL;
}
 
+   /*
+* Mark dra7xx_pcie_msi IRQ as IRQF_NO_THREAD
+* On -RT and if kernel is booting with "threadirqs" cmd line parameter
+* the dra7xx_pcie_msi_irq_handler() will be forced threaded but,
+* in the same time, it's IRQ dispatcher and calls generic_handle_irq(),
+* which, in turn, will be resolved to handle_simple_irq() call.
+* The handle_simple_irq() expected to be called with IRQ disabled, as
+* result kernle will display warning:
+* "irq XXX handler YYY+0x0/0x14 enabled interrupts".
+*/
ret = devm_request_irq(>dev, pp->irq,
-  dra7xx_pcie_msi_irq_handler, IRQF_SHARED,
+  dra7xx_pcie_msi_irq_handler,
+  IRQF_SHARED | IRQF_NO_THREAD,
   "dra7-pcie-msi", pp);
if (ret) {
dev_err(>dev, "failed to request irq\n");
diff --git a/drivers/pci/host/pci-exynos.c b/drivers/pci/host/pci-exynos.c
index 01095e1..d997d22 100644
--- a/drivers/pci/host/pci-exynos.c
+++ b/drivers/pci/host/pci-exynos.c
@@ -522,7 +522,8 @@ static int __init exynos_add_pcie_port(struct pcie_port *pp,
 
ret = devm_request_irq(>dev, pp->msi_irq,
exynos_pcie_msi_irq_handler,
-   IRQF_SHARED, "exynos-pcie", pp);
+   IRQF_SHARED | IRQF_NO_THREAD,
+   "exynos-pcie", pp);
if (ret) {
dev_err(>dev, "failed to request msi irq\n");
return ret;
diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
index 22e8224..9ce7cd1