From: Ian Munsie <imun...@au1.ibm.com> This documentation gives an overview of the hardware architecture, userspace APIs via /dev/cxl/afu0.0 and the syfs files. It also adds a MAINTAINERS file entry for cxl.
Signed-off-by: Ian Munsie <imun...@au1.ibm.com> Signed-off-by: Michael Neuling <mi...@neuling.org> --- Documentation/ABI/testing/sysfs-class-cxl | 142 ++++++++++++ Documentation/ioctl/ioctl-number.txt | 1 + Documentation/powerpc/00-INDEX | 2 + Documentation/powerpc/cxl.txt | 346 ++++++++++++++++++++++++++++++ MAINTAINERS | 7 + 5 files changed, 498 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-class-cxl create mode 100644 Documentation/powerpc/cxl.txt diff --git a/Documentation/ABI/testing/sysfs-class-cxl b/Documentation/ABI/testing/sysfs-class-cxl new file mode 100644 index 0000000..ca429fc --- /dev/null +++ b/Documentation/ABI/testing/sysfs-class-cxl @@ -0,0 +1,142 @@ +Slave contexts (eg. /sys/class/cxl/afu0.0): + +What: /sys/class/cxl/<afu>/irqs_max +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + Maximum number of interrupts that can be requested by userspace. + The default on probe is the maximum that hardware can support + (eg. 2037). Write values will limit userspace applications to + that many userspace interrupts. Must be >= irqs_min. + +What: /sys/class/cxl/<afu>/irqs_min +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + The minimum number of interrupts that userspace must request + on a CXL_START_WORK ioctl. Userspace may omit the + num_interrupts field in the START_WORK IOCTL to get this + minimum automatically. + +What: /sys/class/cxl/<afu>/mmio_size +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + Size of the MMIO space that may be mmaped by userspace. + + +What: /sys/class/cxl/<afu>/models_supported +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + List of the models this AFU supports. + Valid entries are: "dedicated_process" and "afu_directed" + +What: /sys/class/cxl/<afu>/model +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read/write + The current model the AFU is using. Will be one of the models + given in models_supported. Writing will change the model + provided that no user contexts are attached. + + +What: /sys/class/cxl/<afu>/prefault_mode +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read/write + Set the mode for prefaulting in segments into the segment table + when performing the START_WORK ioctl. Possible values: + none: No prefaulting (default) + wed: Treat the wed as an effective address and prefault it + all: all segments this process currently maps + +What: /sys/class/cxl/<afu>/reset +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: write only + Reset the AFU. + +What: /sys/class/cxl/<afu>/api_version +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + List the current version of the kernel/user API. + +What: /sys/class/cxl/<afu>/api_version_com +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + List the lowest version the kernel/user API this + kernel is compatible with. + + + +Master contexts (eg. /sys/class/cxl/afu0.0m) + +What: /sys/class/cxl/<afu>m/mmio_size +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + Size of the MMIO space that may be mmaped by userspace. This + includes all slave contexts space also. + +What: /sys/class/cxl/<afu>m/pp_mmio_len +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + Per Process MMIO space length. + +What: /sys/class/cxl/<afu>m/pp_mmio_off +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + Per Process MMIO space offset. + + +Card info (eg. /sys/class/cxl/card0) + +What: /sys/class/cxl/<card>/caia_version +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + Identifies the CAIA Version the card implements. + +What: /sys/class/cxl/<card>/psl_version +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + Identifies the revision level of the PSL. + +What: /sys/class/cxl/<card>/base_image +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + Identifies the revision level of the base image for devices + that support load-able PSLs. For FPGAs this field identifies + the image contained in the on-adapter flash which is loaded + during the initial program load + +What: /sys/class/cxl/<card>/image_loaded +Date: September 2014 +Contact: Ian Munsie <imun...@au1.ibm.com>, + Michael Neuling <mi...@neuling.org> +Description: read only + Will return "user" or "factory" depending on the image loaded + onto the card + diff --git a/Documentation/ioctl/ioctl-number.txt b/Documentation/ioctl/ioctl-number.txt index 7e240a7..8136e1f 100644 --- a/Documentation/ioctl/ioctl-number.txt +++ b/Documentation/ioctl/ioctl-number.txt @@ -313,6 +313,7 @@ Code Seq#(hex) Include File Comments 0xB1 00-1F PPPoX <mailto:mostr...@styx.uwaterloo.ca> 0xB3 00 linux/mmc/ioctl.h 0xC0 00-0F linux/usb/iowarrior.h +0xCA 00-0F uapi/misc/cxl.h 0xCB 00-1F CBM serial IEC bus in development: <mailto:michael.kl...@puffin.lb.shuttle.de> 0xCD 01 linux/reiserfs_fs.h diff --git a/Documentation/powerpc/00-INDEX b/Documentation/powerpc/00-INDEX index a68784d..116d94d 100644 --- a/Documentation/powerpc/00-INDEX +++ b/Documentation/powerpc/00-INDEX @@ -28,3 +28,5 @@ ptrace.txt - Information on the ptrace interfaces for hardware debug registers. transactional_memory.txt - Overview of the Power8 transactional memory support. +cxl.txt + - Overview of the CXL driver. diff --git a/Documentation/powerpc/cxl.txt b/Documentation/powerpc/cxl.txt new file mode 100644 index 0000000..36f7ba4 --- /dev/null +++ b/Documentation/powerpc/cxl.txt @@ -0,0 +1,346 @@ +Coherent Accelerator Interface (CXL) +==================================== + +Introduction +============ + + The coherent accelerator interface is designed to allow the + coherent connection of FPGA based accelerators (and other devices) + to a POWER system. These devices need to adhere to the Coherent + Accelerator Interface Architecture (CAIA). + + IBM refers to this as the Coherent Accelerator Processor Interface + or CAPI. In the kernel it's referred to by the name CXL to avoid + confusion with the ISDN CAPI subsystem. + +Hardware overview +================= + + POWER8 FPGA + +----------+ +---------+ + | | | | + | CPU | | AFU | + | | | | + | | | | + | | | | + +----------+ +---------+ + | | | | + | CAPP +--------+ PSL | + | | PCIe | | + +----------+ +---------+ + + The POWER8 chip has a Coherently Attached Processor Proxy (CAPP) + unit which is part of the PCIe Host Bridge (PHB). This is managed + by Linux by calls into OPAL. Linux doesn't directly program the + CAPP. + + The FPGA (or coherently attached device) consists of two parts. + The POWER Service Layer (PSL) and the Accelerator Function Unit + (AFU). AFU is used to implement specific functionality behind + the PSL. The PSL, among other things, provides memory address + translation services to allow each AFU direct access to userspace + memory. + + The AFU is the core part of the accelerator (eg. the compression, + crypto etc function). The kernel has no knowledge of the function + of the AFU. Only userspace interacts directly with the AFU. + + The PSL provides the translation and interrupt services that the + AFU needs. This is what the kernel interacts with. For example, + if the AFU needs to read a particular virtual address, it sends + that address to the PSL, the PSL then translates it, fetches the + data from memory and returns it to the AFU. If the PSL has a + translation miss, it interrupts the kernel and the kernel services + the fault. The context to which this fault is serviced is based + on who owns that acceleration function. + +AFU Models +========== + + There are two programming models supported by the AFU. Dedicated + and AFU directed. AFU may support one or both models. + + In dedicated model only one MMU context is supported. In this + model, only one userspace process can use the accelerator at time. + + In AFU directed model, up to 16K simultaneous contexts can be + supported. This means up to 16K simultaneous userspace + applications may use the accelerator (although specific AFUs may + support less). In this mode, the AFU sends a 16 bit context ID + with each of its requests. This tells the PSL which context is + associated with this operation. If the PSL can't translate a + request, the ID can also be accessed by the kernel so it can + determine the associated userspace context to service this + translation with. + +MMIO space +========== + + A portion of the FPGA MMIO space can be directly mapped from the + AFU to userspace. Either the whole space can be mapped (master + context), or just a per context portion (slave context). The + hardware is self describing, hence the kernel can determine the + offset and size of the per context portion. + +Interrupts +========== + + AFUs may generate interrupts that are destined for userspace. These + are received by the kernel as hardware interrupts and passed onto + userspace. + + Data storage faults and error interrupts are handled by the kernel + driver. + +Work Element Descriptor (WED) +============================= + + The WED is a 64bit parameter passed to the AFU when a context is + started. Its format is up to the AFU hence the kernel has no + knowledge of what it represents. Typically it will be a virtual + address pointer to a work queue where the AFU and userspace can + share control and status information or work queues. + + + + +User API +======== + + For AFUs operating in the AFU directed model, the driver will + create two character devices per AFU under /dev/cxl. One for + master and one for slave contexts. + + The master context (eg. /dev/cxl/afu0.0m), has access to all of + the MMIO space that an AFU provides. The slave context + (eg. /dev/cxl/afu0.0) has access to only the per process MMIO + space an AFU provides (AFU directed only). + + For AFUs operating in the dedicated process model, the driver will + only create a single character device per AFU (e.g. + /dev/cxl/afu0.0), which has access to the entire MMIO space that + the AFU provides. + + The following file operations are supported on both slave and + master devices: + + open + + Opens the device and allocates a file descriptor to be used + with the rest of the API. + + A dedicated model AFU only has one context and hence only + allows this device to be opened once. + + An AFU directed model AFU can have many contexts and hence + this device can be opened by as many contexts as available. + + Note: IRQs also need to be allocated per context, which may + also limit the number of contexts that can be allocated, + and hence how many times the device may be opened. The + POWER8 CAPP supports 2040 IRQs and 3 are used by the + kernel, so 2037 are left. If 1 IRQ is needed per + context, then only 2037 contexts can be allocated. If 4 + IRQs are needed per context, then only 2037/4 = 509 + contexts can be allocated. + + ioctl + + CXL_IOCTL_START_WORK: + Starts the AFU context and associates it with the process + memory. Once this ioctl is successfully executed, all + memory mapped into this process is accessible to this AFU + context using the same virtual addresses. No additional + calls are required to map/unmap memory. The AFU memory + context will be updated as userspace allocates and frees + memory. This ioctl returns once the AFU context is + started. + + Takes a pointer to a struct cxl_ioctl_start_work + struct cxl_ioctl_start_work { + __u64 flags; + __u64 wed; + __u64 amr; + __s16 num_interrupts; + __s16 reserved1; + __s32 reserved2; + __u64 reserved3; + __u64 reserved4; + __u64 reserved5; + __u64 reserved6; + }; + + flags: + Indicates which optional fields (e.g. amr, + num_interrupts) in the structure are valid. + + wed: + The Work Element Descriptor (WED) is a 64bit + argument defined by the AFU. Typically this is an + virtual address pointing to an AFU specific + structure describing what work to perform. + + amr: + Authority Mask Register (AMR), same as the powerpc + AMR. + + num_interrupt: + Number of userspace interrupts to request. If not + specified the minimum number required will be + automatically allocated. The min and max number + can be obtained from sysfs. + + reserved fields: + For ABI padding and future extensions + + CXL_IOCTL_GET_PROCESS_ELEMENT: + Get info on current context id. This info is returned + from the kernel as an int. + + Written by the kernel with the context id (AKA process + element) it has allocated. Slave contexts may want to + communicate this to a master process. + + mmap + + An AFU may have a MMIO space to facilitate communication with + the AFU and mmap allows access to this. The size and contents + of this area are specific to the particular AFU. The size can + be discovered via sysfs. + + In the AFU directed model, master contexts will get all of the + MMIO space and slave contexts will get only the per process + space associated with its context. In the dedicated process + model the entire MMIO space is always mapped. + + This mmap call must be done after the IOCTL is started. + + Care should be taken when accessing MMIO space. Only 32 and + 64bit accesses are supported by POWER8. Also, the AFU will be + designed with a specific endian, so all MMIO access should + consider endian (recommend endian(3) variants like: le64toh(), + be64toh() etc). These endian issues equally apply to shared + memory queues the WED may describe. + + read + + Reads an event from the AFU. Will return -EINVAL if the user + supplied buffer to read into is less than 4096 bytes. Blocks + if no events are pending (unless O_NONBLOCK is supplied). Will + return -EIO in the case of an unrecoverable error or if the + card is removed. + + A read may return multiple events. A read will return the + length of the buffer written and it will be a integral number + of events up to the buffer size. Users must supply a buffer + size of at least 4K bytes. + + All events will be return a struct cxl_event which varies in + size. + + struct cxl_event { + struct cxl_event_header header; + union { + struct cxl_event_afu_interrupt irq; + struct cxl_event_data_storage fault; + struct cxl_event_afu_error afu_err; + }; + }; + + A struct cxl_event_header at the start gives: + struct cxl_event_header { + __u16 type; + __u16 size; + __u16 process_element; + __u16 reserved1; + }; + + type: + This gives the type of event. The type determines how + the rest of the event will be structured. These types + are shown below. + + size: + This is the size of the event in bytes including the + header. The start of the next event can be found at + this offset from the start of the current event. + + process_element: + Context ID of the event. Currently this will always + be the current context. Future work may allow + interrupts from one context to be routed to another + (eg. a master contexts handling error interrupts on + behalf of a slave). + + reserved field: + For future extensions and padding. + + If an AFU interrupt event is received, the full structure received is: + struct cxl_event_afu_interrupt { + __u16 flags; + __u16 irq; /* Raised AFU interrupt number */ + __u32 reserved1; + }; + + flags: + These flags indicate which optional fields are present + in this struct. Currently all fields are Mandatory. + + irq: + The IRQ number sent by the AFU. + + reserved field: + For future extensions and padding. + + If a data storage event is received, the full structure received is: + struct cxl_event_data_storage { + __u16 flags; + __u16 reserved1; + __u32 reserved2; + __u64 addr; + __u64 dsisr; + __u64 reserved3; + }; + + flags: + These flags indicate which optional fields are present + in this struct. Currently all fields are Mandatory. + + address: Mandatory + Address of the data storage trying to be accessed by + the AFU. Valid accesses will handled transparently by + the kernel but invalid access will generate this + event. + + dsisr: Manditory + These fields give information on the type of + fault. Copy of the DSISR from PSL hardware when + address fault occured. + + reserved fields: + For future extensions + + If an AFU error event is received, the full structure received is: + struct cxl_event_afu_error { + __u16 flags; + __u16 reserved1; + __u32 reserved2; + __u64 err; + }; + + flags: Mandatory + These flags indicate which optional fields are present + in this struct. Currently all fields are Mandatory. + + err: + Error status from the AFU. AFU defined. + + reserved fields: + For future extensions and padding + +Sysfs Class +=========== + + A cxl sysfs class is added under /sys/class/cxl to facilitate + enumeration and tuning of the accelerators. Its layout is + described in Documentation/ABI/testing/sysfs-class-cxl diff --git a/MAINTAINERS b/MAINTAINERS index 809ecd6..c972be3 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2711,6 +2711,13 @@ W: http://www.chelsio.com S: Supported F: drivers/net/ethernet/chelsio/cxgb4vf/ +CXL (IBM Coherent Accelerator Processor Interface CAPI) DRIVER +M: Ian Munsie <imun...@au1.ibm.com> +M: Michael Neuling <mi...@neuling.org> +L: linuxppc-...@lists.ozlabs.org +S: Supported +F: drivers/misc/cxl/ + STMMAC ETHERNET DRIVER M: Giuseppe Cavallaro <peppe.cavall...@st.com> L: net...@vger.kernel.org -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
