> -----Original Message-----
> From: Jan Kiszka <jan.kis...@siemens.com>
> Sent: Wednesday, August 18, 2021 08:22
> To: Jeroen Van den Keybus <jeroen.vandenkey...@gmail.com>; Reed,
> Scott <sr...@moog.com>
> Cc: xenomai@xenomai.org
> Subject: [EXTERNAL] Re: Question regarding RTnet Ethernet Driver with PCIe
> MSI interrupts
>
> On 17.08.21 22:01, Jeroen Van den Keybus via Xenomai wrote:
> > FWIW, in the past I have worked with e1000 NICs in RTnet that
> > registered their MSI interrupt with the kernel.
> >
> > I do not recall that I had to do anything out of the ordinary.
> >
>
> The main difference was likely that you were on x86 while Scott is dealing
> with an i.MX6-based SoC.
>
> I'm not familiar with the details of MSI routing on that target. If I quickly
> understand arch/arm/boot/dts/imx6qdl.dtsi correctly, all MSIs will be folded
> by the PCI host controller into a single SPI (120), the only interrupt source
> that old SoC understands. In that case, you need to make sure that this SPI is
> handled by I-pipe and, indeed, it treated as muxing IRQ. But I have no idea
> how the demuxing will happen in details. You likely need to study and then
> patch the PCI host controller driver.
>
> Jan
That is correct that I am working with an iMX6-based SoC.
For anyone who might face similar issues, for information here is my patch to
the Linux 4.14.62 PCI host controller (Xenomai 3.0.7 patch already applied).
Regards,
Scott
Subject: [PATCH] Modify PCI MSI interrupts for Xenomai RTDM
Since we have Xenomai RTDM drivers which are "tied" to PCI MSI
interrupts (e.g. Altera TSE driver), we need to also make the PCI MSI
interrupt an RTDM interrupt to avoid latency when the PCI MSI interrupt
is serviced.
Since we have Xenomai RTDM drivers which are "tied" to PCI MSI
interrupts (e.g. Altera TSE driver), we need to first pass the MSI
interrupt to the i-ipipe handler. If there is not an RTDM driver for
the interrupt, then the i-pipe handler calls generic_handle_irq.
---
drivers/pci/dwc/pci-imx6.c | 17 ++++++++++-------
drivers/pci/dwc/pcie-designware-host.c | 2 +-
2 files changed, 11 insertions(+), 8 deletions(-)
diff --git a/drivers/pci/dwc/pci-imx6.c b/drivers/pci/dwc/pci-imx6.c
index b7348353..65eb9ba5 100644
--- a/drivers/pci/dwc/pci-imx6.c
+++ b/drivers/pci/dwc/pci-imx6.c
@@ -33,6 +33,10 @@
#include "pcie-designware.h"
+#include <xenomai/rtdm/driver.h>
+
+static rtdm_irq_t pcie_msi_irq_handle;
+
#define to_imx6_pcie(x)dev_get_drvdata((x)->dev)
enum imx6_pcie_variants {
@@ -545,13 +549,14 @@ static int imx6_pcie_wait_for_speed_change(struct
imx6_pcie *imx6_pcie)
return -EINVAL;
}
-static irqreturn_t imx6_pcie_msi_handler(int irq, void *arg)
+static int imx6_pcie_msi_handler_rtdm(rtdm_irq_t *irq_handle)
{
-struct imx6_pcie *imx6_pcie = arg;
+struct imx6_pcie *imx6_pcie = rtdm_irq_get_arg(irq_handle, struct imx6_pcie);
struct dw_pcie *pci = imx6_pcie->pci;
struct pcie_port *pp = &pci->pp;
-return dw_handle_msi_irq(pp);
+dw_handle_msi_irq(pp);
+return RTDM_IRQ_HANDLED;
}
static int imx6_pcie_establish_link(struct imx6_pcie *imx6_pcie)
@@ -678,10 +683,8 @@ static int imx6_add_pcie_port(struct imx6_pcie *imx6_pcie,
return -ENODEV;
}
-ret = devm_request_irq(dev, pp->msi_irq,
- imx6_pcie_msi_handler,
- IRQF_SHARED | IRQF_NO_THREAD,
- "mx6-pcie-msi", imx6_pcie);
+ret = rtdm_irq_request(&pcie_msi_irq_handle, pp->msi_irq,
imx6_pcie_msi_handler_rtdm,
+RTDM_IRQTYPE_SHARED, "mx6-pcie-msi", imx6_pcie);
if (ret) {
dev_err(dev, "failed to request MSI irq\n");
return ret;
diff --git a/drivers/pci/dwc/pcie-designware-host.c
b/drivers/pci/dwc/pcie-designware-host.c
index a5b8dd07..d0af2cfa 100644
--- a/drivers/pci/dwc/pcie-designware-host.c
+++ b/drivers/pci/dwc/pcie-designware-host.c
@@ -72,7 +72,7 @@ irqreturn_t dw_handle_msi_irq(struct pcie_port *pp)
while ((pos = find_next_bit((unsigned long *) &val, 32,
pos)) != 32) {
irq = irq_find_mapping(pp->irq_domain, i * 32 + pos);
-generic_handle_irq(irq);
+ipipe_handle_demuxed_irq(irq);
dw_pcie_wr_own_conf(pp, PCIE_MSI_INTR0_STATUS + i * 12,
4, 1 << pos);
pos++;
--
2.25.1
>
> >
> > Op di 17 aug. 2021 om 15:37 schreef Reed, Scott via Xenomai <
> > xenomai@xenomai.org>:
> >
> >>> -----Original Message-----
> >>> From: Xenomai <xenomai-boun...@xenomai.org> On Behalf Of Reed,
> Scott
> >>> via Xenomai
> >>> Sent: Thursday, August 5, 2021 19:09
> >>> To: xenomai@xenomai.org
> >>> Subject: [EXTERNAL] Question regarding RTnet Ethernet Driver with
> >>> PCIe
> >> MSI
> >>> interrupts
> >>>
> >>> Hello,
> >>>
> >>> About a year ago, I ported an Ethernet driver we use to be an RTnet
> >>> driver. In general everything is working, but we have run into a
> >>> recent problem which looks like it might be being caused by high
> >>> Ethernet receive interrupt latencies.
> >>>
> >>> The Ethernet interrupts come in as PCIe MSI interrupts and I
> >>> implemented the potentially creative but bad solution of modifying
> >>> the PCIe driver to distribute the MSI interrupts to i-ipipe be
> >>> replacing the call to "generic_handle_irq" with a call to
> >>> "ipipe_handle_demuxed_irq".
> >>>
> >>> I am thinking that this was maybe not the best approach and the
> >>> correct solution would be to modify the PCIe driver so that the
> >>> original MSI interrupt is an RTDM interrupt as if it is not this
> >>> could be causing the high interrupt latencies.
> >>>
> >>> Is my thinking in the right direction or completely off?
> >>
> >> I did not receive any feedback from the forum (maybe I made a mistake
> >> in my post as it was my first post), but in case someone else runs
> >> across this thread and has a similar issue...
> >>
> >> I modified the PCIe driver so that the MSI interrupt is an RTDM
> >> interrupt (i.e. registered interrupt with rtdm_irq_request() as
> >> opposed to with devm_request_irq()) and this solved my problem of
> >> seeing latencies (i.e. delays) in the handling of the Ethernet
> >> receive interrupts (which are packaged in the PCIe MSI interrupt).
> >>
> >> Since my Ethernet receive interrupts (and transmit interrupts) are
> >> also registered as RTDM Interrupts, I needed to keep my original PCIe
> >> driver modification that MSI interrupts are distributed in
> >> dw_handle_msi_irq() using "ipipe_handle_demuxed_irq" instead of
> "generic_handle_irq".
> >>
> >> I assume it would probably also work to register the Ethernet
> >> interrupts as "normal Linux" interrupts and then the this
> >> modification would not be necessary, but I did not verify this and
> >> prefer to have the Ethernet interrupts as RTDM interrupts so the
> >> driver is correct whether or not it is "behind" an PCIe MSI interrupt or
> >> not.
> >>
> >> If anyone would like more details, I can gladly provide them.
> >>
> >> Regards,
> >>
> >> Scott
> >>
> >>>
> >>> Background information
> >>> --------------------------------
> >>> Processor: iMx6 Quad
> >>> Linux 4.14.62
> >>> Xenomai 3.0.7
> >>>
> >>> Thanks in advance for any advice.
> >>>
> >>> Regards,
> >>>
> >>> Scott
> >>
> >>
>
> --
> Siemens AG, T RDA IOT
> Corporate Competence Center Embedded Linux
Hinweis auf Vertraulichkeit: Diese elektronisch übermittelte Nachricht ist nur
für die natürliche oder juristische Person bestimmt, die als Empfänger genannt
ist; sie kann vertrauliche Informationen oder Betriebsgeheimnisse enthalten,
die Eigentum des Absenders sind. Für den Fall, dass Sie nicht der genannte
Empfänger sind, ist Ihnen jede Offenlegung, Verwendung, Vervielfältigung oder
Weiterleitung des Inhalts dieser Informationen streng untersagt. Dasselbe gilt
für alle Handlungen, die sich auf den Inhalt dieser Informationen gründen.
Falls Sie diese Nachricht irrtümlicherweise erhalten haben, bitten wir Sie, den
Absender umgehend per E-Mail darüber in Kenntnis zu setzen und die
ursprüngliche Nachricht zu löschen. Wir danken Ihnen für Ihre Kooperation.
Confidentiality Notice: This electronic mail transmission is intended for the
use of the individual or entity to which it is addressed and may contain
confidential and/or proprietary information belonging to the sender. If you are
not the intended recipient, you are hereby notified that any disclosure, use,
copying, distribution, or the taking of any action in reliance on the contents
of this information is strictly prohibited. If you have received this
transmission in error, please notify the sender immediately by e-mail and
delete the original message. Thank you for your cooperation.
