Re: [Patch Part2 v5 21/31] PCI/MSI: Enhance core to support hierarchy irqdomain
On 11/9/14 14:15, Jiang Liu wrote:
On 2014/11/9 12:26, Suravee Suthikulpanit wrote:
Hi Gerry,
Please see my comments / questions below.
On 11/6/14 21:20, Jiang Liu wrote:
Enhance PCI MSI core to support hierarchy irqdomain, so the common
code could be shared among architectures.
Signed-off-by: Jiang Liu
---
Hi Thomas,
These changes are a temporary solution, I'm working on another
patch set which will refine these interfaces, especially kill
arch_msi_irq_domain_{set|get}_hwirq().
Regards!
Gerry
Would the change includes the struct irqdomain_msi_data proposed by
Thomas here (https://lkml.org/lkml/2014/11/6/210)?
Hi Suravee,
I'm working on another patch set which will refine the way to
create irqdomain for MSI. I hope will solve all issues mentioned by
Thomas. I will post that patch set as RFC within one or two days.
Let me know if I can help.
[...]
+static int msi_domain_alloc(struct irq_domain *domain, unsigned int
virq,
+unsigned int nr_irqs, void *arg)
+{
+int i, ret;
+irq_hw_number_t hwirq = arch_msi_irq_domain_get_hwirq(arg);
+
+if (irq_find_mapping(domain, hwirq) > 0)
+return -EEXIST;
+
+ret = irq_domain_alloc_irqs_parent(domain, virq, nr_irqs, arg);
+if (ret < 0)
+return ret;
+
When executing irq_domain_alloc_irqs_parent(), it triggers the following
warning message due to
WARN_ON(desc->irq_data.chip == &no_irq_chip)
[ cut here ]
WARNING: CPU: 2 PID: 912 at kernel/irq/chip.c:734
__irq_set_handler+0x138/0x13c()
Modules linked in: mlx4_core(+) rtc_efi efivarfs
CPU: 2 PID: 912 Comm: modprobe Not tainted 3.18.0-rc3-p2v5+ #53
Call trace:
[] dump_backtrace+0x0/0x16c
[] show_stack+0x10/0x1c
[] dump_stack+0x74/0x98
[] warn_slowpath_common+0x84/0xac
[] warn_slowpath_null+0x14/0x20
[] __irq_set_handler+0x134/0x13c
[] gic_irq_domain_map+0x4c/0xac
[] gic_irq_domain_alloc+0x60/0x88
[] gicv2m_domain_alloc+0x30/0xa8
[] __irq_domain_alloc_irqs+0x144/0x30c
[] gicv2m_setup_msi_irq+0xc0/0x118
[] arch_setup_msi_irq+0x34/0x60
[] arch_setup_msi_irqs+0x50/0xb0
[] pci_enable_msix+0x310/0x39c
[] pci_enable_msix_range+0x34/0x9c
However, I think if we call irq_domain_set_hwirq_and_chip() in the
msi_create_irq_domain() as suggested by Thomas, that should also fix
this issue.
We didn't trigger this warning on x86 because only the outer-most
irqdomain will set irq flow handler. On ARM, the inner GIC domain
will set irq flow handler, but the irq_data->chip for outer-most
domain hasn't been set yet.
I'm still find a way out here. Maybe we need to relax the WARN_ON().
What if we just switch the order and call the
irq_domain_alloc_irqs_parent() after the loop? That way we can still
keep the WARN_ON().
Suravee
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [Patch Part2 v5 21/31] PCI/MSI: Enhance core to support hierarchy irqdomain
On 2014/11/9 12:26, Suravee Suthikulpanit wrote:
> Hi Gerry,
>
> Please see my comments / questions below.
>
> On 11/6/14 21:20, Jiang Liu wrote:
>> Enhance PCI MSI core to support hierarchy irqdomain, so the common
>> code could be shared among architectures.
>>
>> Signed-off-by: Jiang Liu
>> ---
>> Hi Thomas,
>> These changes are a temporary solution, I'm working on another
>> patch set which will refine these interfaces, especially kill
>> arch_msi_irq_domain_{set|get}_hwirq().
>> Regards!
>> Gerry
>
> Would the change includes the struct irqdomain_msi_data proposed by
> Thomas here (https://lkml.org/lkml/2014/11/6/210)?
Hi Suravee,
I'm working on another patch set which will refine the way to
create irqdomain for MSI. I hope will solve all issues mentioned by
Thomas. I will post that patch set as RFC within one or two days.
>
>> [...]
>> +static int msi_domain_alloc(struct irq_domain *domain, unsigned int
>> virq,
>> +unsigned int nr_irqs, void *arg)
>> +{
>> +int i, ret;
>> +irq_hw_number_t hwirq = arch_msi_irq_domain_get_hwirq(arg);
>> +
>> +if (irq_find_mapping(domain, hwirq) > 0)
>> +return -EEXIST;
>> +
>> +ret = irq_domain_alloc_irqs_parent(domain, virq, nr_irqs, arg);
>> +if (ret < 0)
>> +return ret;
>> +
>
> When executing irq_domain_alloc_irqs_parent(), it triggers the following
> warning message due to
>
> WARN_ON(desc->irq_data.chip == &no_irq_chip)
>
> [ cut here ]
> WARNING: CPU: 2 PID: 912 at kernel/irq/chip.c:734
> __irq_set_handler+0x138/0x13c()
> Modules linked in: mlx4_core(+) rtc_efi efivarfs
> CPU: 2 PID: 912 Comm: modprobe Not tainted 3.18.0-rc3-p2v5+ #53
> Call trace:
> [] dump_backtrace+0x0/0x16c
> [] show_stack+0x10/0x1c
> [] dump_stack+0x74/0x98
> [] warn_slowpath_common+0x84/0xac
> [] warn_slowpath_null+0x14/0x20
> [] __irq_set_handler+0x134/0x13c
> [] gic_irq_domain_map+0x4c/0xac
> [] gic_irq_domain_alloc+0x60/0x88
> [] gicv2m_domain_alloc+0x30/0xa8
> [] __irq_domain_alloc_irqs+0x144/0x30c
> [] gicv2m_setup_msi_irq+0xc0/0x118
> [] arch_setup_msi_irq+0x34/0x60
> [] arch_setup_msi_irqs+0x50/0xb0
> [] pci_enable_msix+0x310/0x39c
> [] pci_enable_msix_range+0x34/0x9c
>
>
> However, I think if we call irq_domain_set_hwirq_and_chip() in the
> msi_create_irq_domain() as suggested by Thomas, that should also fix
> this issue.
We didn't trigger this warning on x86 because only the outer-most
irqdomain will set irq flow handler. On ARM, the inner GIC domain
will set irq flow handler, but the irq_data->chip for outer-most
domain hasn't been set yet.
I'm still find a way out here. Maybe we need to relax the WARN_ON().
>
>> +for (i = 0; i < nr_irqs; i++) {
>> +irq_domain_set_hwirq_and_chip(domain, virq + i, hwirq + i,
>> + domain->host_data,
>> + (void *)(long)i);
>> +__irq_set_handler(virq + i, handle_edge_irq, 0, "edge");
>> +}
>
> Is there are a way to specify other type of handler besides edge?
Yes, the new patch set will address this requirement.
>
>> [...]
>> +static void msi_domain_activate(struct irq_domain *domain,
>> +struct irq_data *irq_data)
>> +{
>> +struct msi_msg msg;
>> +
>> +/*
>> + * irq_data->chip_data is MSI/MSI-X offset.
>> + * MSI-X message is written per-IRQ, the offset is always 0.
>> + * MSI message denotes a contiguous group of IRQs, written for
>> 0th IRQ.
>> + */
>> +if (irq_data->chip_data)
>> +return;
>
> Actually, I am a bit confused with this comment here. If you look at
> "/drivers/pci/msi.c: arch_setup_msi_irq()", it calls
> irq_set_chip_data(des->irq, chip) where chip is *msi_chip.
>
> It looks like if the arch uses this API, it would conflict with what you
> have here where the irq_data->chip_data would be not NULL at the logic
> above, and would always return.
Good point, it's work on x86, but will break ARM or other platforms.
Will refine this part.
Regards!
Gerry
>
> Thank you,
>
> Suravee
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [Patch Part2 v5 21/31] PCI/MSI: Enhance core to support hierarchy irqdomain
Hi Gerry,
Please see my comments / questions below.
On 11/6/14 21:20, Jiang Liu wrote:
Enhance PCI MSI core to support hierarchy irqdomain, so the common
code could be shared among architectures.
Signed-off-by: Jiang Liu
---
Hi Thomas,
These changes are a temporary solution, I'm working on another
patch set which will refine these interfaces, especially kill
arch_msi_irq_domain_{set|get}_hwirq().
Regards!
Gerry
Would the change includes the struct irqdomain_msi_data proposed by
Thomas here (https://lkml.org/lkml/2014/11/6/210)?
[...]
+static int msi_domain_alloc(struct irq_domain *domain, unsigned int virq,
+ unsigned int nr_irqs, void *arg)
+{
+ int i, ret;
+ irq_hw_number_t hwirq = arch_msi_irq_domain_get_hwirq(arg);
+
+ if (irq_find_mapping(domain, hwirq) > 0)
+ return -EEXIST;
+
+ ret = irq_domain_alloc_irqs_parent(domain, virq, nr_irqs, arg);
+ if (ret < 0)
+ return ret;
+
When executing irq_domain_alloc_irqs_parent(), it triggers the following
warning message due to
WARN_ON(desc->irq_data.chip == &no_irq_chip)
[ cut here ]
WARNING: CPU: 2 PID: 912 at kernel/irq/chip.c:734
__irq_set_handler+0x138/0x13c()
Modules linked in: mlx4_core(+) rtc_efi efivarfs
CPU: 2 PID: 912 Comm: modprobe Not tainted 3.18.0-rc3-p2v5+ #53
Call trace:
[] dump_backtrace+0x0/0x16c
[] show_stack+0x10/0x1c
[] dump_stack+0x74/0x98
[] warn_slowpath_common+0x84/0xac
[] warn_slowpath_null+0x14/0x20
[] __irq_set_handler+0x134/0x13c
[] gic_irq_domain_map+0x4c/0xac
[] gic_irq_domain_alloc+0x60/0x88
[] gicv2m_domain_alloc+0x30/0xa8
[] __irq_domain_alloc_irqs+0x144/0x30c
[] gicv2m_setup_msi_irq+0xc0/0x118
[] arch_setup_msi_irq+0x34/0x60
[] arch_setup_msi_irqs+0x50/0xb0
[] pci_enable_msix+0x310/0x39c
[] pci_enable_msix_range+0x34/0x9c
However, I think if we call irq_domain_set_hwirq_and_chip() in the
msi_create_irq_domain() as suggested by Thomas, that should also fix
this issue.
+ for (i = 0; i < nr_irqs; i++) {
+ irq_domain_set_hwirq_and_chip(domain, virq + i, hwirq + i,
+ domain->host_data,
+ (void *)(long)i);
+ __irq_set_handler(virq + i, handle_edge_irq, 0, "edge");
+ }
Is there are a way to specify other type of handler besides edge?
[...]
+static void msi_domain_activate(struct irq_domain *domain,
+ struct irq_data *irq_data)
+{
+ struct msi_msg msg;
+
+ /*
+* irq_data->chip_data is MSI/MSI-X offset.
+* MSI-X message is written per-IRQ, the offset is always 0.
+* MSI message denotes a contiguous group of IRQs, written for 0th IRQ.
+*/
+ if (irq_data->chip_data)
+ return;
Actually, I am a bit confused with this comment here. If you look at
"/drivers/pci/msi.c: arch_setup_msi_irq()", it calls
irq_set_chip_data(des->irq, chip) where chip is *msi_chip.
It looks like if the arch uses this API, it would conflict with what you
have here where the irq_data->chip_data would be not NULL at the logic
above, and would always return.
Thank you,
Suravee
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [Patch Part2 v5 21/31] PCI/MSI: Enhance core to support hierarchy irqdomain
On Thu, 6 Nov 2014, Jiang Liu wrote:
> Enhance PCI MSI core to support hierarchy irqdomain, so the common
> code could be shared among architectures.
>
> Signed-off-by: Jiang Liu
> ---
> Hi Thomas,
> These changes are a temporary solution, I'm working on another
> patch set which will refine these interfaces, especially kill
> arch_msi_irq_domain_{set|get}_hwirq().
Ok. Please don't forget the irq_chip change :)
Thanks,
tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Patch Part2 v5 21/31] PCI/MSI: Enhance core to support hierarchy irqdomain
Enhance PCI MSI core to support hierarchy irqdomain, so the common
code could be shared among architectures.
Signed-off-by: Jiang Liu
---
Hi Thomas,
These changes are a temporary solution, I'm working on another
patch set which will refine these interfaces, especially kill
arch_msi_irq_domain_{set|get}_hwirq().
Regards!
Gerry
---
drivers/pci/Kconfig |4 ++
drivers/pci/msi.c | 134 +++
include/linux/msi.h | 14 ++
3 files changed, 152 insertions(+)
diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index b9db0f2ce11f..022e89745f86 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -16,6 +16,10 @@ config PCI_MSI
If you don't know what to do here, say Y.
+config PCI_MSI_IRQ_DOMAIN
+ bool
+ depends on PCI_MSI && IRQ_DOMAIN_HIERARCHY
+
config PCI_DEBUG
bool "PCI Debugging"
depends on PCI && DEBUG_KERNEL
diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index da181c59394b..8de7c8774fd2 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -19,6 +19,7 @@
#include
#include
#include
+#include
#include "pci.h"
@@ -1098,3 +1099,136 @@ int pci_enable_msix_range(struct pci_dev *dev, struct
msix_entry *entries,
return nvec;
}
EXPORT_SYMBOL(pci_enable_msix_range);
+
+#ifdef CONFIG_PCI_MSI_IRQ_DOMAIN
+/*
+ * Generate a unique ID number for each possible MSI source, the ID number
+ * is only used within the irqdomain.
+ */
+static inline irq_hw_number_t
+msi_get_hwirq(struct pci_dev *dev, struct msi_desc *desc)
+{
+ return (irq_hw_number_t)desc->msi_attrib.entry_nr |
+ PCI_DEVID(dev->bus->number, dev->devfn) << 11 |
+ (pci_domain_nr(dev->bus) & 0x) << 27;
+}
+
+static int msi_domain_alloc(struct irq_domain *domain, unsigned int virq,
+ unsigned int nr_irqs, void *arg)
+{
+ int i, ret;
+ irq_hw_number_t hwirq = arch_msi_irq_domain_get_hwirq(arg);
+
+ if (irq_find_mapping(domain, hwirq) > 0)
+ return -EEXIST;
+
+ ret = irq_domain_alloc_irqs_parent(domain, virq, nr_irqs, arg);
+ if (ret < 0)
+ return ret;
+
+ for (i = 0; i < nr_irqs; i++) {
+ irq_domain_set_hwirq_and_chip(domain, virq + i, hwirq + i,
+ domain->host_data,
+ (void *)(long)i);
+ __irq_set_handler(virq + i, handle_edge_irq, 0, "edge");
+ }
+
+ return ret;
+}
+
+static void msi_domain_free(struct irq_domain *domain, unsigned int virq,
+ unsigned int nr_irqs)
+{
+ int i;
+
+ for (i = 0; i < nr_irqs; i++) {
+ struct msi_desc *desc = irq_get_msi_desc(virq);
+
+ if (desc)
+ desc->irq = 0;
+ }
+ irq_domain_free_irqs_top(domain, virq, nr_irqs);
+}
+
+static void msi_domain_activate(struct irq_domain *domain,
+ struct irq_data *irq_data)
+{
+ struct msi_msg msg;
+
+ /*
+* irq_data->chip_data is MSI/MSI-X offset.
+* MSI-X message is written per-IRQ, the offset is always 0.
+* MSI message denotes a contiguous group of IRQs, written for 0th IRQ.
+*/
+ if (irq_data->chip_data)
+ return;
+
+ BUG_ON(irq_chip_compose_msi_msg(irq_data, &msg));
+ __write_msi_msg(irq_data->msi_desc, &msg);
+}
+
+static void msi_domain_deactivate(struct irq_domain *domain,
+ struct irq_data *irq_data)
+{
+ struct msi_msg msg;
+
+ if (!irq_data->chip_data) {
+ memset(&msg, 0, sizeof(msg));
+ __write_msi_msg(irq_data->msi_desc, &msg);
+ }
+}
+
+static struct irq_domain_ops msi_domain_ops = {
+ .alloc = msi_domain_alloc,
+ .free = msi_domain_free,
+ .activate = msi_domain_activate,
+ .deactivate = msi_domain_deactivate,
+};
+
+struct irq_domain *msi_create_irq_domain(struct device_node *of_node,
+struct irq_chip *chip,
+struct irq_domain *parent)
+{
+ struct irq_domain *domain;
+
+ domain = irq_domain_add_tree(of_node, &msi_domain_ops, chip);
+ if (!domain)
+ return NULL;
+
+ domain->parent = parent;
+
+ return domain;
+}
+
+int msi_irq_domain_alloc_irqs(struct irq_domain *domain, int type,
+ struct pci_dev *dev, void *arg)
+{
+ int i, virq;
+ struct msi_desc *desc;
+ int node = dev_to_node(&dev->dev);
+
+ list_for_each_entry(desc, &dev->msi_list, list) {
+ arch_msi_irq_domain_set_hwirq(arg, msi_get_hwirq(dev, desc));
+ virq = irq_domain_alloc_irqs(domain, desc->nvec_used,
+node, arg);
+ if (virq < 0) {
+ /*
