On 3/16/21 1:29 PM, Max Zhen wrote: > Hi Tom, > > > On 2/26/21 7:01 AM, Tom Rix wrote: >> CAUTION: This message has originated from an External Source. Please use >> proper judgment and caution when opening attachments, clicking links, or >> responding to this email. >> >> >> A question i do not know the answer to. >> >> Seems like 'golden' is linked to a manufacturing (diagnostics?) image. >> >> If the public will never see it, should handling it here be done ? >> >> Moritz, do you know ? > > > Golden image is preloaded on the device when it is shipped to customer. Then, > customer can load other shells (from Xilinx or some other vendor). If > something goes wrong with the shell, customer can always go back to golden > and start over again. So, golden image is going to be used in public, not > just internally by Xilinx. > > Thanks for the explanation.
>> >> >> On 2/17/21 10:40 PM, Lizhi Hou wrote: >>> The PCIE device driver which attaches to management function on Alveo >> to the management > > > Sure. > > >>> devices. It instantiates one or more partition drivers which in turn >> more fpga partition / group ? > > > Group driver. > > >>> instantiate platform drivers. The instantiation of partition and platform >>> drivers is completely data driven. >> data driven ? everything is data driven. do you mean dtb driven ? > > > Data driven means not hard-coded. Here data means meta data which is > presented in device tree format, dtb. > > >>> Signed-off-by: Sonal Santan <sonal.san...@xilinx.com> >>> Signed-off-by: Max Zhen <max.z...@xilinx.com> >>> Signed-off-by: Lizhi Hou <liz...@xilinx.com> >>> --- >>> drivers/fpga/xrt/include/xroot.h | 114 +++++++++++ >>> drivers/fpga/xrt/mgmt/root.c | 342 +++++++++++++++++++++++++++++++ >>> 2 files changed, 456 insertions(+) >>> create mode 100644 drivers/fpga/xrt/include/xroot.h >>> create mode 100644 drivers/fpga/xrt/mgmt/root.c >>> >>> diff --git a/drivers/fpga/xrt/include/xroot.h >>> b/drivers/fpga/xrt/include/xroot.h >>> new file mode 100644 >>> index 000000000000..752e10daa85e >>> --- /dev/null >>> +++ b/drivers/fpga/xrt/include/xroot.h >>> @@ -0,0 +1,114 @@ >>> +/* SPDX-License-Identifier: GPL-2.0 */ >>> +/* >>> + * Header file for Xilinx Runtime (XRT) driver >>> + * >>> + * Copyright (C) 2020-2021 Xilinx, Inc. >>> + * >>> + * Authors: >>> + * Cheng Zhen <m...@xilinx.com> >>> + */ >>> + >>> +#ifndef _XRT_ROOT_H_ >>> +#define _XRT_ROOT_H_ >>> + >>> +#include <linux/pci.h> >>> +#include "subdev_id.h" >>> +#include "events.h" >>> + >>> +typedef bool (*xrt_subdev_match_t)(enum xrt_subdev_id, >>> + struct platform_device *, void *); >>> +#define XRT_SUBDEV_MATCH_PREV ((xrt_subdev_match_t)-1) >>> +#define XRT_SUBDEV_MATCH_NEXT ((xrt_subdev_match_t)-2) >>> + >>> +/* >>> + * Root IOCTL calls. >>> + */ >>> +enum xrt_root_ioctl_cmd { >>> + /* Leaf actions. */ >>> + XRT_ROOT_GET_LEAF = 0, >>> + XRT_ROOT_PUT_LEAF, >>> + XRT_ROOT_GET_LEAF_HOLDERS, >>> + >>> + /* Group actions. */ >>> + XRT_ROOT_CREATE_GROUP, >>> + XRT_ROOT_REMOVE_GROUP, >>> + XRT_ROOT_LOOKUP_GROUP, >>> + XRT_ROOT_WAIT_GROUP_BRINGUP, >>> + >>> + /* Event actions. */ >>> + XRT_ROOT_EVENT, >> should this be XRT_ROOT_EVENT_SYNC ? > > > Sure. > > >>> + XRT_ROOT_EVENT_ASYNC, >>> + >>> + /* Device info. */ >>> + XRT_ROOT_GET_RESOURCE, >>> + XRT_ROOT_GET_ID, >>> + >>> + /* Misc. */ >>> + XRT_ROOT_HOT_RESET, >>> + XRT_ROOT_HWMON, >>> +}; >>> + >>> +struct xrt_root_ioctl_get_leaf { >>> + struct platform_device *xpigl_pdev; /* caller's pdev */ >> xpigl_ ? unneeded suffix in element names > > > It's needed since the it might be included and used in > 1 .c files. I'd like > to keep it's name unique. This is an element name, the variable name sound be unique enough to make it clear. This is not a critical issue, ok as-is. > > >>> + xrt_subdev_match_t xpigl_match_cb; >>> + void *xpigl_match_arg; >>> + struct platform_device *xpigl_leaf; /* target leaf pdev */ >>> +}; >>> + >>> +struct xrt_root_ioctl_put_leaf { >>> + struct platform_device *xpipl_pdev; /* caller's pdev */ >>> + struct platform_device *xpipl_leaf; /* target's pdev */ >> caller_pdev; >> >> target_pdev; > > > Sure. > > >> >>> +}; >>> + >>> +struct xrt_root_ioctl_lookup_group { >>> + struct platform_device *xpilp_pdev; /* caller's pdev */ >>> + xrt_subdev_match_t xpilp_match_cb; >>> + void *xpilp_match_arg; >>> + int xpilp_grp_inst; >>> +}; >>> + >>> +struct xrt_root_ioctl_get_holders { >>> + struct platform_device *xpigh_pdev; /* caller's pdev */ >>> + char *xpigh_holder_buf; >>> + size_t xpigh_holder_buf_len; >>> +}; >>> + >>> +struct xrt_root_ioctl_get_res { >>> + struct resource *xpigr_res; >>> +}; >>> + >>> +struct xrt_root_ioctl_get_id { >>> + unsigned short xpigi_vendor_id; >>> + unsigned short xpigi_device_id; >>> + unsigned short xpigi_sub_vendor_id; >>> + unsigned short xpigi_sub_device_id; >>> +}; >>> + >>> +struct xrt_root_ioctl_hwmon { >>> + bool xpih_register; >>> + const char *xpih_name; >>> + void *xpih_drvdata; >>> + const struct attribute_group **xpih_groups; >>> + struct device *xpih_hwmon_dev; >>> +}; >>> + >>> +typedef int (*xrt_subdev_root_cb_t)(struct device *, void *, u32, void *); >> This function pointer type is important, please add a comment about its use >> and expected parameters > > > Added. > > >>> +int xrt_subdev_root_request(struct platform_device *self, u32 cmd, void >>> *arg); >>> + >>> +/* >>> + * Defines physical function (MPF / UPF) specific operations >>> + * needed in common root driver. >>> + */ >>> +struct xroot_pf_cb { >>> + void (*xpc_hot_reset)(struct pci_dev *pdev); >> This is only ever set to xmgmt_root_hot_reset, why is this abstraction >> needed ? > > > As comment says, hot reset is implemented differently in MPF and UPF driver. > So, we need this callback in this common code. Note that we have not added > UPF code in our initial patch yet. It will be added in the future. > > >>> +}; >>> + >>> +int xroot_probe(struct pci_dev *pdev, struct xroot_pf_cb *cb, void **root); >>> +void xroot_remove(void *root); >>> +bool xroot_wait_for_bringup(void *root); >>> +int xroot_add_vsec_node(void *root, char *dtb); >>> +int xroot_create_group(void *xr, char *dtb); >>> +int xroot_add_simple_node(void *root, char *dtb, const char *endpoint); >>> +void xroot_broadcast(void *root, enum xrt_events evt); >>> + >>> +#endif /* _XRT_ROOT_H_ */ >>> diff --git a/drivers/fpga/xrt/mgmt/root.c b/drivers/fpga/xrt/mgmt/root.c >>> new file mode 100644 >>> index 000000000000..583a37c9d30c >>> --- /dev/null >>> +++ b/drivers/fpga/xrt/mgmt/root.c >>> @@ -0,0 +1,342 @@ >>> +// SPDX-License-Identifier: GPL-2.0 >>> +/* >>> + * Xilinx Alveo Management Function Driver >>> + * >>> + * Copyright (C) 2020-2021 Xilinx, Inc. >>> + * >>> + * Authors: >>> + * Cheng Zhen <m...@xilinx.com> >>> + */ >>> + >>> +#include <linux/module.h> >>> +#include <linux/pci.h> >>> +#include <linux/aer.h> >>> +#include <linux/vmalloc.h> >>> +#include <linux/delay.h> >>> + >>> +#include "xroot.h" >>> +#include "main-impl.h" >>> +#include "metadata.h" >>> + >>> +#define XMGMT_MODULE_NAME "xmgmt" >> The xrt modules would be more easily identified with a 'xrt' prefix instead >> of 'x' > > > We will change the module name to xrt-mgmt. > > >>> +#define XMGMT_DRIVER_VERSION "4.0.0" >>> + >>> +#define XMGMT_PDEV(xm) ((xm)->pdev) >>> +#define XMGMT_DEV(xm) (&(XMGMT_PDEV(xm)->dev)) >>> +#define xmgmt_err(xm, fmt, args...) \ >>> + dev_err(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args) >>> +#define xmgmt_warn(xm, fmt, args...) \ >>> + dev_warn(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args) >>> +#define xmgmt_info(xm, fmt, args...) \ >>> + dev_info(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args) >>> +#define xmgmt_dbg(xm, fmt, args...) \ >>> + dev_dbg(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args) >>> +#define XMGMT_DEV_ID(_pcidev) \ >>> + ({ typeof(_pcidev) (pcidev) = (_pcidev); \ >>> + ((pci_domain_nr((pcidev)->bus) << 16) | \ >>> + PCI_DEVID((pcidev)->bus->number, 0)); }) >>> + >>> +static struct class *xmgmt_class; >>> +static const struct pci_device_id xmgmt_pci_ids[] = { >>> + { PCI_DEVICE(0x10EE, 0xd020), }, /* Alveo U50 (golden image) */ >>> + { PCI_DEVICE(0x10EE, 0x5020), }, /* Alveo U50 */ >> demagic this table, look at dfl-pci for how to use existing #define for the >> vendor and create a new on for the device. If there are vf's add them at >> the same time. >> >> What is a golden image ? > > > Fixed. Please see my comments above for golden image. > > >> >>> + { 0, } >>> +}; >>> + >>> +struct xmgmt { >>> + struct pci_dev *pdev; >>> + void *root; >>> + >>> + bool ready; >>> +}; >>> + >>> +static int xmgmt_config_pci(struct xmgmt *xm) >>> +{ >>> + struct pci_dev *pdev = XMGMT_PDEV(xm); >>> + int rc; >>> + >>> + rc = pcim_enable_device(pdev); >>> + if (rc < 0) { >>> + xmgmt_err(xm, "failed to enable device: %d", rc); >>> + return rc; >>> + } >>> + >>> + rc = pci_enable_pcie_error_reporting(pdev); >>> + if (rc) >>> + xmgmt_warn(xm, "failed to enable AER: %d", rc); >>> + >>> + pci_set_master(pdev); >>> + >>> + rc = pcie_get_readrq(pdev); >> Review this call, it does not go negative > > > I'll remove the check against negative value. > > >>> + if (rc < 0) { >>> + xmgmt_err(xm, "failed to read mrrs %d", rc); >>> + return rc; >>> + } >> this is a quirk, add a comment. > > > Will remove. > > >>> + if (rc > 512) { >>> + rc = pcie_set_readrq(pdev, 512); >>> + if (rc) { >>> + xmgmt_err(xm, "failed to force mrrs %d", rc); >> similar calls do not fail here. > > > Will remove. > > >>> + return rc; >>> + } >>> + } >>> + >>> + return 0; >>> +} >>> + >>> +static int xmgmt_match_slot_and_save(struct device *dev, void *data) >>> +{ >>> + struct xmgmt *xm = data; >>> + struct pci_dev *pdev = to_pci_dev(dev); >>> + >>> + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) { >>> + pci_cfg_access_lock(pdev); >>> + pci_save_state(pdev); >>> + } >>> + >>> + return 0; >>> +} >>> + >>> +static void xmgmt_pci_save_config_all(struct xmgmt *xm) >>> +{ >>> + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_save); >> This is a bus call, not a device call. >> >> Can this be changed into something like what hot reset does ? > > > We are working on both mgmt pf and user pf here, so sort of like a bus. But, > it might be better to refactor this when we have our own bus type > implementation. We do not need to make PCIE bus call. We will fix this in V5 > patch set where we'll implement our own bus type. > > ok >> >>> +} >>> + >>> +static int xmgmt_match_slot_and_restore(struct device *dev, void *data) >>> +{ >>> + struct xmgmt *xm = data; >>> + struct pci_dev *pdev = to_pci_dev(dev); >>> + >>> + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) { >>> + pci_restore_state(pdev); >>> + pci_cfg_access_unlock(pdev); >>> + } >>> + >>> + return 0; >>> +} >>> + >>> +static void xmgmt_pci_restore_config_all(struct xmgmt *xm) >>> +{ >>> + bus_for_each_dev(&pci_bus_type, NULL, xm, >>> xmgmt_match_slot_and_restore); >>> +} >>> + >>> +static void xmgmt_root_hot_reset(struct pci_dev *pdev) >>> +{ >>> + struct xmgmt *xm = pci_get_drvdata(pdev); >>> + struct pci_bus *bus; >>> + u8 pci_bctl; >>> + u16 pci_cmd, devctl; >>> + int i, ret; >>> + >>> + xmgmt_info(xm, "hot reset start"); >>> + >>> + xmgmt_pci_save_config_all(xm); >>> + >>> + pci_disable_device(pdev); >>> + >>> + bus = pdev->bus; >>> + >>> + /* >>> + * When flipping the SBR bit, device can fall off the bus. This is >>> + * usually no problem at all so long as drivers are working properly >>> + * after SBR. However, some systems complain bitterly when the device >>> + * falls off the bus. >>> + * The quick solution is to temporarily disable the SERR reporting of >>> + * switch port during SBR. >>> + */ >>> + >>> + pci_read_config_word(bus->self, PCI_COMMAND, &pci_cmd); >>> + pci_write_config_word(bus->self, PCI_COMMAND, >>> + (pci_cmd & ~PCI_COMMAND_SERR)); >>> + pcie_capability_read_word(bus->self, PCI_EXP_DEVCTL, &devctl); >>> + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, >>> + (devctl & ~PCI_EXP_DEVCTL_FERE)); >>> + pci_read_config_byte(bus->self, PCI_BRIDGE_CONTROL, &pci_bctl); >>> + pci_bctl |= PCI_BRIDGE_CTL_BUS_RESET; >>> + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl); >> how the pci config values are set and cleared should be consistent. >> >> this call should be >> >> pci_write_config_byte (... pci_bctl | PCI_BRIDGE_CTL_BUF_RESET ) >> >> and the next &= avoided > > > Sure. > > >> >>> + >>> + msleep(100); >>> + pci_bctl &= ~PCI_BRIDGE_CTL_BUS_RESET; >>> + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl); >>> + ssleep(1); >>> + >>> + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, devctl); >>> + pci_write_config_word(bus->self, PCI_COMMAND, pci_cmd); >>> + >>> + ret = pci_enable_device(pdev); >>> + if (ret) >>> + xmgmt_err(xm, "failed to enable device, ret %d", ret); >>> + >>> + for (i = 0; i < 300; i++) { >>> + pci_read_config_word(pdev, PCI_COMMAND, &pci_cmd); >>> + if (pci_cmd != 0xffff) >> what happens with i == 300 and pci_cmd is still 0xffff ? > > > Something wrong happens to the device since it's not coming back after the > reset. In this case, the device cannot be used and the only way to recover is > to power cycle the system so that the shell can be reloaded from the flash on > the device. > > so check and add a dev_crit() to let the user know. >>> + break; >>> + msleep(20); >>> + } >>> + >>> + xmgmt_info(xm, "waiting for %d ms", i * 20); >>> + xmgmt_pci_restore_config_all(xm); >>> + xmgmt_config_pci(xm); >>> +} >>> + >>> +static int xmgmt_create_root_metadata(struct xmgmt *xm, char **root_dtb) >>> +{ >>> + char *dtb = NULL; >>> + int ret; >>> + >>> + ret = xrt_md_create(XMGMT_DEV(xm), &dtb); >>> + if (ret) { >>> + xmgmt_err(xm, "create metadata failed, ret %d", ret); >>> + goto failed; >>> + } >>> + >>> + ret = xroot_add_vsec_node(xm->root, dtb); >>> + if (ret == -ENOENT) { >>> + /* >>> + * We may be dealing with a MFG board. >>> + * Try vsec-golden which will bring up all hard-coded leaves >>> + * at hard-coded offsets. >>> + */ >>> + ret = xroot_add_simple_node(xm->root, dtb, >>> XRT_MD_NODE_VSEC_GOLDEN); >> Manufacturing diagnostics ? > > > This is for golden image support. Please see my comments above. Ok, i got it :) Thanks, looking forward next rev Tom > > > Thanks, > > Max > >> >> Tom >> >>> + } else if (ret == 0) { >>> + ret = xroot_add_simple_node(xm->root, dtb, >>> XRT_MD_NODE_MGMT_MAIN); >>> + } >>> + if (ret) >>> + goto failed; >>> + >>> + *root_dtb = dtb; >>> + return 0; >>> + >>> +failed: >>> + vfree(dtb); >>> + return ret; >>> +} >>> + >>> +static ssize_t ready_show(struct device *dev, >>> + struct device_attribute *da, >>> + char *buf) >>> +{ >>> + struct pci_dev *pdev = to_pci_dev(dev); >>> + struct xmgmt *xm = pci_get_drvdata(pdev); >>> + >>> + return sprintf(buf, "%d\n", xm->ready); >>> +} >>> +static DEVICE_ATTR_RO(ready); >>> + >>> +static struct attribute *xmgmt_root_attrs[] = { >>> + &dev_attr_ready.attr, >>> + NULL >>> +}; >>> + >>> +static struct attribute_group xmgmt_root_attr_group = { >>> + .attrs = xmgmt_root_attrs, >>> +}; >>> + >>> +static struct xroot_pf_cb xmgmt_xroot_pf_cb = { >>> + .xpc_hot_reset = xmgmt_root_hot_reset, >>> +}; >>> + >>> +static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id >>> *id) >>> +{ >>> + int ret; >>> + struct device *dev = &pdev->dev; >>> + struct xmgmt *xm = devm_kzalloc(dev, sizeof(*xm), GFP_KERNEL); >>> + char *dtb = NULL; >>> + >>> + if (!xm) >>> + return -ENOMEM; >>> + xm->pdev = pdev; >>> + pci_set_drvdata(pdev, xm); >>> + >>> + ret = xmgmt_config_pci(xm); >>> + if (ret) >>> + goto failed; >>> + >>> + ret = xroot_probe(pdev, &xmgmt_xroot_pf_cb, &xm->root); >>> + if (ret) >>> + goto failed; >>> + >>> + ret = xmgmt_create_root_metadata(xm, &dtb); >>> + if (ret) >>> + goto failed_metadata; >>> + >>> + ret = xroot_create_group(xm->root, dtb); >>> + vfree(dtb); >>> + if (ret) >>> + xmgmt_err(xm, "failed to create root group: %d", ret); >>> + >>> + if (!xroot_wait_for_bringup(xm->root)) >>> + xmgmt_err(xm, "failed to bringup all groups"); >>> + else >>> + xm->ready = true; >>> + >>> + ret = sysfs_create_group(&pdev->dev.kobj, &xmgmt_root_attr_group); >>> + if (ret) { >>> + /* Warning instead of failing the probe. */ >>> + xmgmt_warn(xm, "create xmgmt root attrs failed: %d", ret); >>> + } >>> + >>> + xroot_broadcast(xm->root, XRT_EVENT_POST_CREATION); >>> + xmgmt_info(xm, "%s started successfully", XMGMT_MODULE_NAME); >>> + return 0; >>> + >>> +failed_metadata: >>> + (void)xroot_remove(xm->root); >>> +failed: >>> + pci_set_drvdata(pdev, NULL); >>> + return ret; >>> +} >>> + >>> +static void xmgmt_remove(struct pci_dev *pdev) >>> +{ >>> + struct xmgmt *xm = pci_get_drvdata(pdev); >>> + >>> + xroot_broadcast(xm->root, XRT_EVENT_PRE_REMOVAL); >>> + sysfs_remove_group(&pdev->dev.kobj, &xmgmt_root_attr_group); >>> + (void)xroot_remove(xm->root); >>> + pci_disable_pcie_error_reporting(xm->pdev); >>> + xmgmt_info(xm, "%s cleaned up successfully", XMGMT_MODULE_NAME); >>> +} >>> + >>> +static struct pci_driver xmgmt_driver = { >>> + .name = XMGMT_MODULE_NAME, >>> + .id_table = xmgmt_pci_ids, >>> + .probe = xmgmt_probe, >>> + .remove = xmgmt_remove, >>> +}; >>> + >>> +static int __init xmgmt_init(void) >>> +{ >>> + int res = 0; >>> + >>> + res = xmgmt_main_register_leaf(); >>> + if (res) >>> + return res; >>> + >>> + xmgmt_class = class_create(THIS_MODULE, XMGMT_MODULE_NAME); >>> + if (IS_ERR(xmgmt_class)) >>> + return PTR_ERR(xmgmt_class); >>> + >>> + res = pci_register_driver(&xmgmt_driver); >>> + if (res) { >>> + class_destroy(xmgmt_class); >>> + return res; >>> + } >>> + >>> + return 0; >>> +} >>> + >>> +static __exit void xmgmt_exit(void) >>> +{ >>> + pci_unregister_driver(&xmgmt_driver); >>> + class_destroy(xmgmt_class); >>> + xmgmt_main_unregister_leaf(); >>> +} >>> + >>> +module_init(xmgmt_init); >>> +module_exit(xmgmt_exit); >>> + >>> +MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids); >>> +MODULE_VERSION(XMGMT_DRIVER_VERSION); >>> +MODULE_AUTHOR("XRT Team <runt...@xilinx.com>"); >>> +MODULE_DESCRIPTION("Xilinx Alveo management function driver"); >>> +MODULE_LICENSE("GPL v2"); >