Re: [EXT] Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On 21-02-04 21:53:29, John Groves (jgroves) wrote: >Micron Confidential > > > >From: Dan Williams >Date: Monday, February 1, 2021 at 1:28 PM >To: Ben Widawsky >Cc: Konrad Rzeszutek Wilk , >linux-...@vger.kernel.org , Linux ACPI >, Linux Kernel Mailing List >, linux-nvdimm >, Linux PCI , >Bjorn Helgaas , Chris Browy >, Christoph Hellwig , Ira >Weiny , Jon Masters , Jonathan >Cameron , Rafael Wysocki >, Randy Dunlap , >Vishal Verma , daniel@alibaba-inc.com >, John Groves (jgroves) > , Kelley, Sean V >Subject: [EXT] Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox > >CAUTION: EXTERNAL EMAIL. Do not click links or open attachments unless >you recognize the sender and were expecting this message. >On Mon, Feb 1, 2021 at 11:13 AM Ben Widawsky >wrote: >> >> On 21-02-01 12:54:00, Konrad Rzeszutek Wilk wrote: >> > > +#define >cxl_doorbell_busy(cxlm) >\ >> > > + (cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET) >&\ >> > > +CXLDEV_MB_CTRL_DOORBELL) >> > > + >> > > +#define CXL_MAILBOX_TIMEOUT_US 2000 >> > >> > You been using the spec for the values. Is that number also from it >? >> > >> >> Yes it is. I'll add a comment with the spec reference. > > > >From section 8.2.8.4 in the CXL 2.0 spec: “The mailbox command timeout >is 2 seconds.” So this should be: > > >#define CXL_MAILBOX_TIMEOUT_US 200 > > >…right? 2000us is 2ms… > > Thanks. This was caught already in earlier review by David Rientjes It's renamed CXL_MAILBOX_TIMEOUT_MS 2000
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On 21-02-02 15:54:03, Dan Williams wrote: > On Tue, Feb 2, 2021 at 2:57 PM Ben Widawsky wrote: > > > > On 21-02-01 12:00:18, Dan Williams wrote: > > > On Sat, Jan 30, 2021 at 3:52 PM David Rientjes > > > wrote: > > > > > > > > On Fri, 29 Jan 2021, Ben Widawsky wrote: > > > > > > > > > Provide enough functionality to utilize the mailbox of a memory > > > > > device. > > > > > The mailbox is used to interact with the firmware running on the > > > > > memory > > > > > device. > > > > > > > > > > The CXL specification defines separate capabilities for the mailbox > > > > > and > > > > > the memory device. The mailbox interface has a doorbell to indicate > > > > > ready to accept commands and the memory device has a capability > > > > > register > > > > > that indicates the mailbox interface is ready. The expectation is that > > > > > the doorbell-ready is always later than the memory-device-indication > > > > > that the mailbox is ready. > > > > > > > > > > Create a function to handle sending a command, optionally with a > > > > > payload, to the memory device, polling on a result, and then > > > > > optionally > > > > > copying out the payload. The algorithm for doing this comes straight > > > > > out > > > > > of the CXL 2.0 specification. > > > > > > > > > > Primary mailboxes are capable of generating an interrupt when > > > > > submitting > > > > > a command in the background. That implementation is saved for a later > > > > > time. > > > > > > > > > > Secondary mailboxes aren't implemented at this time. > > > > > > > > > > The flow is proven with one implemented command, "identify". Because > > > > > the > > > > > class code has already told the driver this is a memory device and the > > > > > identify command is mandatory. > > > > > > > > > > Signed-off-by: Ben Widawsky > > > > > --- > > > > > drivers/cxl/Kconfig | 14 ++ > > > > > drivers/cxl/cxl.h | 39 + > > > > > drivers/cxl/mem.c | 342 > > > > > +++- > > > > > 3 files changed, 394 insertions(+), 1 deletion(-) > > > > > > > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > > > > index 3b66b46af8a0..fe591f74af96 100644 > > > > > --- a/drivers/cxl/Kconfig > > > > > +++ b/drivers/cxl/Kconfig > > > > > @@ -32,4 +32,18 @@ config CXL_MEM > > > > > Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification. > > > > > > > > > > If unsure say 'm'. > > > > > + > > > > > +config CXL_MEM_INSECURE_DEBUG > > > > > + bool "CXL.mem debugging" > > > > > + depends on CXL_MEM > > > > > + help > > > > > + Enable debug of all CXL command payloads. > > > > > + > > > > > + Some CXL devices and controllers support encryption and other > > > > > + security features. The payloads for the commands that enable > > > > > + those features may contain sensitive clear-text security > > > > > + material. Disable debug of those command payloads by default. > > > > > + If you are a kernel developer actively working on CXL > > > > > + security enabling say Y, otherwise say N. > > > > > > > > Not specific to this patch, but the reference to encryption made me > > > > curious about integrity: are all CXL.mem devices compatible with DIMP? > > > > Some? None? > > > > > > The encryption here is "device passphrase" similar to the NVDIMM > > > Security Management described here: > > > > > > https://pmem.io/documents/IntelOptanePMem_DSM_Interface-V2.0.pdf > > > > > > The LIBNVDIMM enabling wrapped this support with the Linux keys > > > interface which among other things enforces wrapping the clear text > > > passphrase with a Linux "trusted/encrypted" key. > > > > > > Additionally, the CXL.io interface optionally supports PCI IDE: > > > > > > https://www.intel.com/content/dam/www/public/us/en/documents/reference-guides/pcie-device-security-enhancements.pdf > > > > > > I'm otherwise not familiar with the DIMP acronym? > > > > > > > > + > > > > > endif > > > > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > > > > > index a3da7f8050c4..df3d97154b63 100644 > > > > > --- a/drivers/cxl/cxl.h > > > > > +++ b/drivers/cxl/cxl.h > > > > > @@ -31,9 +31,36 @@ > > > > > #define CXLDEV_MB_CAPS_OFFSET 0x00 > > > > > #define CXLDEV_MB_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0) > > > > > #define CXLDEV_MB_CTRL_OFFSET 0x04 > > > > > +#define CXLDEV_MB_CTRL_DOORBELL BIT(0) > > > > > #define CXLDEV_MB_CMD_OFFSET 0x08 > > > > > +#define CXLDEV_MB_CMD_COMMAND_OPCODE_MASK GENMASK(15, 0) > > > > > +#define CXLDEV_MB_CMD_PAYLOAD_LENGTH_MASK GENMASK(36, 16) > > > > > #define CXLDEV_MB_STATUS_OFFSET 0x10 > > > > > +#define CXLDEV_MB_STATUS_RET_CODE_MASK GENMASK(47, 32) > > > > > #define CXLDEV_MB_BG_CMD_STATUS_OFFSET 0x18 > > > > > +#define CXLDEV_MB_PAYLOAD_OFFSET 0x20 > > > > > + > > > > > +/* Memory Device (CXL 2.0 - 8.2.8.5.1.1) */ > > > > > +#define CXLMDEV_STATUS_OFFSET 0x0 > > > > > +#define CXLMDEV_DEV_FATAL BIT(0) > > > > > +#define
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On Tue, Feb 2, 2021 at 2:57 PM Ben Widawsky wrote: > > On 21-02-01 12:00:18, Dan Williams wrote: > > On Sat, Jan 30, 2021 at 3:52 PM David Rientjes wrote: > > > > > > On Fri, 29 Jan 2021, Ben Widawsky wrote: > > > > > > > Provide enough functionality to utilize the mailbox of a memory device. > > > > The mailbox is used to interact with the firmware running on the memory > > > > device. > > > > > > > > The CXL specification defines separate capabilities for the mailbox and > > > > the memory device. The mailbox interface has a doorbell to indicate > > > > ready to accept commands and the memory device has a capability register > > > > that indicates the mailbox interface is ready. The expectation is that > > > > the doorbell-ready is always later than the memory-device-indication > > > > that the mailbox is ready. > > > > > > > > Create a function to handle sending a command, optionally with a > > > > payload, to the memory device, polling on a result, and then optionally > > > > copying out the payload. The algorithm for doing this comes straight out > > > > of the CXL 2.0 specification. > > > > > > > > Primary mailboxes are capable of generating an interrupt when submitting > > > > a command in the background. That implementation is saved for a later > > > > time. > > > > > > > > Secondary mailboxes aren't implemented at this time. > > > > > > > > The flow is proven with one implemented command, "identify". Because the > > > > class code has already told the driver this is a memory device and the > > > > identify command is mandatory. > > > > > > > > Signed-off-by: Ben Widawsky > > > > --- > > > > drivers/cxl/Kconfig | 14 ++ > > > > drivers/cxl/cxl.h | 39 + > > > > drivers/cxl/mem.c | 342 +++- > > > > 3 files changed, 394 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > > > index 3b66b46af8a0..fe591f74af96 100644 > > > > --- a/drivers/cxl/Kconfig > > > > +++ b/drivers/cxl/Kconfig > > > > @@ -32,4 +32,18 @@ config CXL_MEM > > > > Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification. > > > > > > > > If unsure say 'm'. > > > > + > > > > +config CXL_MEM_INSECURE_DEBUG > > > > + bool "CXL.mem debugging" > > > > + depends on CXL_MEM > > > > + help > > > > + Enable debug of all CXL command payloads. > > > > + > > > > + Some CXL devices and controllers support encryption and other > > > > + security features. The payloads for the commands that enable > > > > + those features may contain sensitive clear-text security > > > > + material. Disable debug of those command payloads by default. > > > > + If you are a kernel developer actively working on CXL > > > > + security enabling say Y, otherwise say N. > > > > > > Not specific to this patch, but the reference to encryption made me > > > curious about integrity: are all CXL.mem devices compatible with DIMP? > > > Some? None? > > > > The encryption here is "device passphrase" similar to the NVDIMM > > Security Management described here: > > > > https://pmem.io/documents/IntelOptanePMem_DSM_Interface-V2.0.pdf > > > > The LIBNVDIMM enabling wrapped this support with the Linux keys > > interface which among other things enforces wrapping the clear text > > passphrase with a Linux "trusted/encrypted" key. > > > > Additionally, the CXL.io interface optionally supports PCI IDE: > > > > https://www.intel.com/content/dam/www/public/us/en/documents/reference-guides/pcie-device-security-enhancements.pdf > > > > I'm otherwise not familiar with the DIMP acronym? > > > > > > + > > > > endif > > > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > > > > index a3da7f8050c4..df3d97154b63 100644 > > > > --- a/drivers/cxl/cxl.h > > > > +++ b/drivers/cxl/cxl.h > > > > @@ -31,9 +31,36 @@ > > > > #define CXLDEV_MB_CAPS_OFFSET 0x00 > > > > #define CXLDEV_MB_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0) > > > > #define CXLDEV_MB_CTRL_OFFSET 0x04 > > > > +#define CXLDEV_MB_CTRL_DOORBELL BIT(0) > > > > #define CXLDEV_MB_CMD_OFFSET 0x08 > > > > +#define CXLDEV_MB_CMD_COMMAND_OPCODE_MASK GENMASK(15, 0) > > > > +#define CXLDEV_MB_CMD_PAYLOAD_LENGTH_MASK GENMASK(36, 16) > > > > #define CXLDEV_MB_STATUS_OFFSET 0x10 > > > > +#define CXLDEV_MB_STATUS_RET_CODE_MASK GENMASK(47, 32) > > > > #define CXLDEV_MB_BG_CMD_STATUS_OFFSET 0x18 > > > > +#define CXLDEV_MB_PAYLOAD_OFFSET 0x20 > > > > + > > > > +/* Memory Device (CXL 2.0 - 8.2.8.5.1.1) */ > > > > +#define CXLMDEV_STATUS_OFFSET 0x0 > > > > +#define CXLMDEV_DEV_FATAL BIT(0) > > > > +#define CXLMDEV_FW_HALT BIT(1) > > > > +#define CXLMDEV_STATUS_MEDIA_STATUS_MASK GENMASK(3, 2) > > > > +#define CXLMDEV_MS_NOT_READY 0 > > > > +#define CXLMDEV_MS_READY 1 > > > > +#define CXLMDEV_MS_ERROR 2 > > > > +#define CXLMDEV_MS_DISABLED 3 > > > > +#define CXLMDEV_READY(status) \ > > > > + (CXL_GET_FIELD(status,
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On 21-02-01 12:00:18, Dan Williams wrote: > On Sat, Jan 30, 2021 at 3:52 PM David Rientjes wrote: > > > > On Fri, 29 Jan 2021, Ben Widawsky wrote: > > > > > Provide enough functionality to utilize the mailbox of a memory device. > > > The mailbox is used to interact with the firmware running on the memory > > > device. > > > > > > The CXL specification defines separate capabilities for the mailbox and > > > the memory device. The mailbox interface has a doorbell to indicate > > > ready to accept commands and the memory device has a capability register > > > that indicates the mailbox interface is ready. The expectation is that > > > the doorbell-ready is always later than the memory-device-indication > > > that the mailbox is ready. > > > > > > Create a function to handle sending a command, optionally with a > > > payload, to the memory device, polling on a result, and then optionally > > > copying out the payload. The algorithm for doing this comes straight out > > > of the CXL 2.0 specification. > > > > > > Primary mailboxes are capable of generating an interrupt when submitting > > > a command in the background. That implementation is saved for a later > > > time. > > > > > > Secondary mailboxes aren't implemented at this time. > > > > > > The flow is proven with one implemented command, "identify". Because the > > > class code has already told the driver this is a memory device and the > > > identify command is mandatory. > > > > > > Signed-off-by: Ben Widawsky > > > --- > > > drivers/cxl/Kconfig | 14 ++ > > > drivers/cxl/cxl.h | 39 + > > > drivers/cxl/mem.c | 342 +++- > > > 3 files changed, 394 insertions(+), 1 deletion(-) > > > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > > index 3b66b46af8a0..fe591f74af96 100644 > > > --- a/drivers/cxl/Kconfig > > > +++ b/drivers/cxl/Kconfig > > > @@ -32,4 +32,18 @@ config CXL_MEM > > > Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification. > > > > > > If unsure say 'm'. > > > + > > > +config CXL_MEM_INSECURE_DEBUG > > > + bool "CXL.mem debugging" > > > + depends on CXL_MEM > > > + help > > > + Enable debug of all CXL command payloads. > > > + > > > + Some CXL devices and controllers support encryption and other > > > + security features. The payloads for the commands that enable > > > + those features may contain sensitive clear-text security > > > + material. Disable debug of those command payloads by default. > > > + If you are a kernel developer actively working on CXL > > > + security enabling say Y, otherwise say N. > > > > Not specific to this patch, but the reference to encryption made me > > curious about integrity: are all CXL.mem devices compatible with DIMP? > > Some? None? > > The encryption here is "device passphrase" similar to the NVDIMM > Security Management described here: > > https://pmem.io/documents/IntelOptanePMem_DSM_Interface-V2.0.pdf > > The LIBNVDIMM enabling wrapped this support with the Linux keys > interface which among other things enforces wrapping the clear text > passphrase with a Linux "trusted/encrypted" key. > > Additionally, the CXL.io interface optionally supports PCI IDE: > > https://www.intel.com/content/dam/www/public/us/en/documents/reference-guides/pcie-device-security-enhancements.pdf > > I'm otherwise not familiar with the DIMP acronym? > > > > + > > > endif > > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > > > index a3da7f8050c4..df3d97154b63 100644 > > > --- a/drivers/cxl/cxl.h > > > +++ b/drivers/cxl/cxl.h > > > @@ -31,9 +31,36 @@ > > > #define CXLDEV_MB_CAPS_OFFSET 0x00 > > > #define CXLDEV_MB_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0) > > > #define CXLDEV_MB_CTRL_OFFSET 0x04 > > > +#define CXLDEV_MB_CTRL_DOORBELL BIT(0) > > > #define CXLDEV_MB_CMD_OFFSET 0x08 > > > +#define CXLDEV_MB_CMD_COMMAND_OPCODE_MASK GENMASK(15, 0) > > > +#define CXLDEV_MB_CMD_PAYLOAD_LENGTH_MASK GENMASK(36, 16) > > > #define CXLDEV_MB_STATUS_OFFSET 0x10 > > > +#define CXLDEV_MB_STATUS_RET_CODE_MASK GENMASK(47, 32) > > > #define CXLDEV_MB_BG_CMD_STATUS_OFFSET 0x18 > > > +#define CXLDEV_MB_PAYLOAD_OFFSET 0x20 > > > + > > > +/* Memory Device (CXL 2.0 - 8.2.8.5.1.1) */ > > > +#define CXLMDEV_STATUS_OFFSET 0x0 > > > +#define CXLMDEV_DEV_FATAL BIT(0) > > > +#define CXLMDEV_FW_HALT BIT(1) > > > +#define CXLMDEV_STATUS_MEDIA_STATUS_MASK GENMASK(3, 2) > > > +#define CXLMDEV_MS_NOT_READY 0 > > > +#define CXLMDEV_MS_READY 1 > > > +#define CXLMDEV_MS_ERROR 2 > > > +#define CXLMDEV_MS_DISABLED 3 > > > +#define CXLMDEV_READY(status) \ > > > + (CXL_GET_FIELD(status, CXLMDEV_STATUS_MEDIA_STATUS) == > > > CXLMDEV_MS_READY) > > > +#define CXLMDEV_MBOX_IF_READY BIT(4) > > > +#define CXLMDEV_RESET_NEEDED_SHIFT 5 > > > +#define CXLMDEV_RESET_NEEDED_MASK GENMASK(7, 5) > > > +#define CXLMDEV_RESET_NEEDED_NOT 0 > > > +#define
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On 21-01-30 15:51:57, David Rientjes wrote:
> On Fri, 29 Jan 2021, Ben Widawsky wrote:
>
[snip]
> > +/**
> > + * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory device.
> > + * @cxlm: The CXL memory device to communicate with.
> > + * @mbox_cmd: Command to send to the memory device.
> > + *
> > + * Context: Any context. Expects mbox_lock to be held.
> > + * Return: -ETIMEDOUT if timeout occurred waiting for completion. 0 on
> > success.
> > + * Caller should check the return code in @mbox_cmd to make sure it
> > + * succeeded.
> > + *
> > + * This is a generic form of the CXL mailbox send command, thus the only
> > I/O
> > + * operations used are cxl_read_mbox_reg(). Memory devices, and perhaps
> > other
> > + * types of CXL devices may have further information available upon error
> > + * conditions.
> > + *
> > + * The CXL spec allows for up to two mailboxes. The intention is for the
> > primary
> > + * mailbox to be OS controlled and the secondary mailbox to be used by
> > system
> > + * firmware. This allows the OS and firmware to communicate with the
> > device and
> > + * not need to coordinate with each other. The driver only uses the primary
> > + * mailbox.
> > + */
> > +static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
> > +struct mbox_cmd *mbox_cmd)
> > +{
> > + void __iomem *payload = cxlm->mbox.regs + CXLDEV_MB_PAYLOAD_OFFSET;
>
> Do you need to verify the payload is non-empty per 8.2.8.4?
>
What do you mean exactly? Emptiness or lack thereof is what determines the size
parameter of the mailbox interface (if we want to input data, we need to write
size, if we got output data, we have to read size bytes out of the payload
registers).
Perhaps I miss the point though, could you elaborate?
[snip]
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On Sat, Jan 30, 2021 at 3:52 PM David Rientjes wrote: > > On Fri, 29 Jan 2021, Ben Widawsky wrote: > > > Provide enough functionality to utilize the mailbox of a memory device. > > The mailbox is used to interact with the firmware running on the memory > > device. > > > > The CXL specification defines separate capabilities for the mailbox and > > the memory device. The mailbox interface has a doorbell to indicate > > ready to accept commands and the memory device has a capability register > > that indicates the mailbox interface is ready. The expectation is that > > the doorbell-ready is always later than the memory-device-indication > > that the mailbox is ready. > > > > Create a function to handle sending a command, optionally with a > > payload, to the memory device, polling on a result, and then optionally > > copying out the payload. The algorithm for doing this comes straight out > > of the CXL 2.0 specification. > > > > Primary mailboxes are capable of generating an interrupt when submitting > > a command in the background. That implementation is saved for a later > > time. > > > > Secondary mailboxes aren't implemented at this time. > > > > The flow is proven with one implemented command, "identify". Because the > > class code has already told the driver this is a memory device and the > > identify command is mandatory. > > > > Signed-off-by: Ben Widawsky > > --- > > drivers/cxl/Kconfig | 14 ++ > > drivers/cxl/cxl.h | 39 + > > drivers/cxl/mem.c | 342 +++- > > 3 files changed, 394 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > index 3b66b46af8a0..fe591f74af96 100644 > > --- a/drivers/cxl/Kconfig > > +++ b/drivers/cxl/Kconfig > > @@ -32,4 +32,18 @@ config CXL_MEM > > Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification. > > > > If unsure say 'm'. > > + > > +config CXL_MEM_INSECURE_DEBUG > > + bool "CXL.mem debugging" > > + depends on CXL_MEM > > + help > > + Enable debug of all CXL command payloads. > > + > > + Some CXL devices and controllers support encryption and other > > + security features. The payloads for the commands that enable > > + those features may contain sensitive clear-text security > > + material. Disable debug of those command payloads by default. > > + If you are a kernel developer actively working on CXL > > + security enabling say Y, otherwise say N. > > Not specific to this patch, but the reference to encryption made me > curious about integrity: are all CXL.mem devices compatible with DIMP? > Some? None? The encryption here is "device passphrase" similar to the NVDIMM Security Management described here: https://pmem.io/documents/IntelOptanePMem_DSM_Interface-V2.0.pdf The LIBNVDIMM enabling wrapped this support with the Linux keys interface which among other things enforces wrapping the clear text passphrase with a Linux "trusted/encrypted" key. Additionally, the CXL.io interface optionally supports PCI IDE: https://www.intel.com/content/dam/www/public/us/en/documents/reference-guides/pcie-device-security-enhancements.pdf I'm otherwise not familiar with the DIMP acronym? > > + > > endif > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > > index a3da7f8050c4..df3d97154b63 100644 > > --- a/drivers/cxl/cxl.h > > +++ b/drivers/cxl/cxl.h > > @@ -31,9 +31,36 @@ > > #define CXLDEV_MB_CAPS_OFFSET 0x00 > > #define CXLDEV_MB_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0) > > #define CXLDEV_MB_CTRL_OFFSET 0x04 > > +#define CXLDEV_MB_CTRL_DOORBELL BIT(0) > > #define CXLDEV_MB_CMD_OFFSET 0x08 > > +#define CXLDEV_MB_CMD_COMMAND_OPCODE_MASK GENMASK(15, 0) > > +#define CXLDEV_MB_CMD_PAYLOAD_LENGTH_MASK GENMASK(36, 16) > > #define CXLDEV_MB_STATUS_OFFSET 0x10 > > +#define CXLDEV_MB_STATUS_RET_CODE_MASK GENMASK(47, 32) > > #define CXLDEV_MB_BG_CMD_STATUS_OFFSET 0x18 > > +#define CXLDEV_MB_PAYLOAD_OFFSET 0x20 > > + > > +/* Memory Device (CXL 2.0 - 8.2.8.5.1.1) */ > > +#define CXLMDEV_STATUS_OFFSET 0x0 > > +#define CXLMDEV_DEV_FATAL BIT(0) > > +#define CXLMDEV_FW_HALT BIT(1) > > +#define CXLMDEV_STATUS_MEDIA_STATUS_MASK GENMASK(3, 2) > > +#define CXLMDEV_MS_NOT_READY 0 > > +#define CXLMDEV_MS_READY 1 > > +#define CXLMDEV_MS_ERROR 2 > > +#define CXLMDEV_MS_DISABLED 3 > > +#define CXLMDEV_READY(status) \ > > + (CXL_GET_FIELD(status, CXLMDEV_STATUS_MEDIA_STATUS) == > > CXLMDEV_MS_READY) > > +#define CXLMDEV_MBOX_IF_READY BIT(4) > > +#define CXLMDEV_RESET_NEEDED_SHIFT 5 > > +#define CXLMDEV_RESET_NEEDED_MASK GENMASK(7, 5) > > +#define CXLMDEV_RESET_NEEDED_NOT 0 > > +#define CXLMDEV_RESET_NEEDED_COLD 1 > > +#define CXLMDEV_RESET_NEEDED_WARM 2 > > +#define CXLMDEV_RESET_NEEDED_HOT 3 > > +#define CXLMDEV_RESET_NEEDED_CXL 4 > > +#define CXLMDEV_RESET_NEEDED(status) \ > > + (CXL_GET_FIELD(status, CXLMDEV_RESET_NEEDED) != > >
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On Mon, Feb 1, 2021 at 11:13 AM Ben Widawsky wrote:
>
> On 21-02-01 12:54:00, Konrad Rzeszutek Wilk wrote:
> > > +#define cxl_doorbell_busy(cxlm)
> > > \
> > > + (cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET) &
> > > \
> > > +CXLDEV_MB_CTRL_DOORBELL)
> > > +
> > > +#define CXL_MAILBOX_TIMEOUT_US 2000
> >
> > You been using the spec for the values. Is that number also from it ?
> >
>
> Yes it is. I'll add a comment with the spec reference.
>
> > > +
> > > +enum opcode {
> > > + CXL_MBOX_OP_IDENTIFY= 0x4000,
> > > + CXL_MBOX_OP_MAX = 0x1
> > > +};
> > > +
> > > +/**
> > > + * struct mbox_cmd - A command to be submitted to hardware.
> > > + * @opcode: (input) The command set and command submitted to hardware.
> > > + * @payload_in: (input) Pointer to the input payload.
> > > + * @payload_out: (output) Pointer to the output payload. Must be
> > > allocated by
> > > + * the caller.
> > > + * @size_in: (input) Number of bytes to load from @payload.
> > > + * @size_out: (output) Number of bytes loaded into @payload.
> > > + * @return_code: (output) Error code returned from hardware.
> > > + *
> > > + * This is the primary mechanism used to send commands to the hardware.
> > > + * All the fields except @payload_* correspond exactly to the fields
> > > described in
> > > + * Command Register section of the CXL 2.0 spec (8.2.8.4.5). @payload_in
> > > and
> > > + * @payload_out are written to, and read from the Command Payload
> > > Registers
> > > + * defined in (8.2.8.4.8).
> > > + */
> > > +struct mbox_cmd {
> > > + u16 opcode;
> > > + void *payload_in;
> > > + void *payload_out;
> >
> > On a 32-bit OS (not that we use those that more, but lets assume
> > someone really wants to), the void is 4-bytes, while on 64-bit it is
> > 8-bytes.
> >
> > `pahole` is your friend as I think there is a gap between opcode and
> > payload_in in the structure.
> >
> > > + size_t size_in;
> > > + size_t size_out;
> >
> > And those can also change depending on 32-bit/64-bit.
> >
> > > + u16 return_code;
> > > +#define CXL_MBOX_SUCCESS 0
> > > +};
> >
> > Do you want to use __packed to match with the spec?
> >
> > Ah, reading later you don't care about it.
> >
> > In that case may I recommend you move 'return_code' (or perhaps just
> > call it rc?) to be right after opcode? Less of gaps in that structure.
> >
>
> I guess I hadn't realized we're supposed to try to fully pack structs by
> default.
This is just the internal parsed context of a command, I can't imagine
packing is relevant here. pahole optimization feels premature as well.
>
> > > +
> > > +static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> > > +{
> > > + const int timeout = msecs_to_jiffies(CXL_MAILBOX_TIMEOUT_US);
> > > + const unsigned long start = jiffies;
> > > + unsigned long end = start;
> > > +
> > > + while (cxl_doorbell_busy(cxlm)) {
> > > + end = jiffies;
> > > +
> > > + if (time_after(end, start + timeout)) {
> > > + /* Check again in case preempted before timeout test
> > > */
> > > + if (!cxl_doorbell_busy(cxlm))
> > > + break;
> > > + return -ETIMEDOUT;
> > > + }
> > > + cpu_relax();
> > > + }
> >
> > Hm, that is not very scheduler friendly. I mean we are sitting here for
> > 2000us (2 ms) - that is quite the amount of time spinning.
> >
> > Should this perhaps be put in a workqueue?
>
> So let me first point you to the friendlier version which was shot down:
> https://lore.kernel.org/linux-cxl/2020054356.793390-8-ben.widaw...@intel.com/
>
> I'm not opposed to this being moved to a workqueue at some point, but I think
> that's unnecessary complexity currently. The reality is that it's expected
> that
> commands will finish way sooner than this or be implemented as background
> commands. I've heard a person who makes a lot of the spec decisions say, "if
> it's 2 seconds, nobody will use these things".
That said, asynchronous probe needs to be enabled for the next driver update.
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On 21-02-01 12:54:00, Konrad Rzeszutek Wilk wrote:
> > +#define cxl_doorbell_busy(cxlm)
> > \
> > + (cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET) &\
> > +CXLDEV_MB_CTRL_DOORBELL)
> > +
> > +#define CXL_MAILBOX_TIMEOUT_US 2000
>
> You been using the spec for the values. Is that number also from it ?
>
Yes it is. I'll add a comment with the spec reference.
> > +
> > +enum opcode {
> > + CXL_MBOX_OP_IDENTIFY= 0x4000,
> > + CXL_MBOX_OP_MAX = 0x1
> > +};
> > +
> > +/**
> > + * struct mbox_cmd - A command to be submitted to hardware.
> > + * @opcode: (input) The command set and command submitted to hardware.
> > + * @payload_in: (input) Pointer to the input payload.
> > + * @payload_out: (output) Pointer to the output payload. Must be allocated
> > by
> > + * the caller.
> > + * @size_in: (input) Number of bytes to load from @payload.
> > + * @size_out: (output) Number of bytes loaded into @payload.
> > + * @return_code: (output) Error code returned from hardware.
> > + *
> > + * This is the primary mechanism used to send commands to the hardware.
> > + * All the fields except @payload_* correspond exactly to the fields
> > described in
> > + * Command Register section of the CXL 2.0 spec (8.2.8.4.5). @payload_in
> > and
> > + * @payload_out are written to, and read from the Command Payload Registers
> > + * defined in (8.2.8.4.8).
> > + */
> > +struct mbox_cmd {
> > + u16 opcode;
> > + void *payload_in;
> > + void *payload_out;
>
> On a 32-bit OS (not that we use those that more, but lets assume
> someone really wants to), the void is 4-bytes, while on 64-bit it is
> 8-bytes.
>
> `pahole` is your friend as I think there is a gap between opcode and
> payload_in in the structure.
>
> > + size_t size_in;
> > + size_t size_out;
>
> And those can also change depending on 32-bit/64-bit.
>
> > + u16 return_code;
> > +#define CXL_MBOX_SUCCESS 0
> > +};
>
> Do you want to use __packed to match with the spec?
>
> Ah, reading later you don't care about it.
>
> In that case may I recommend you move 'return_code' (or perhaps just
> call it rc?) to be right after opcode? Less of gaps in that structure.
>
I guess I hadn't realized we're supposed to try to fully pack structs by
default.
> > +
> > +static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> > +{
> > + const int timeout = msecs_to_jiffies(CXL_MAILBOX_TIMEOUT_US);
> > + const unsigned long start = jiffies;
> > + unsigned long end = start;
> > +
> > + while (cxl_doorbell_busy(cxlm)) {
> > + end = jiffies;
> > +
> > + if (time_after(end, start + timeout)) {
> > + /* Check again in case preempted before timeout test */
> > + if (!cxl_doorbell_busy(cxlm))
> > + break;
> > + return -ETIMEDOUT;
> > + }
> > + cpu_relax();
> > + }
>
> Hm, that is not very scheduler friendly. I mean we are sitting here for
> 2000us (2 ms) - that is quite the amount of time spinning.
>
> Should this perhaps be put in a workqueue?
So let me first point you to the friendlier version which was shot down:
https://lore.kernel.org/linux-cxl/2020054356.793390-8-ben.widaw...@intel.com/
I'm not opposed to this being moved to a workqueue at some point, but I think
that's unnecessary complexity currently. The reality is that it's expected that
commands will finish way sooner than this or be implemented as background
commands. I've heard a person who makes a lot of the spec decisions say, "if
it's 2 seconds, nobody will use these things".
I think adding the summary of this back and forth as a comment to the existing
code makes a lot of sense.
> > +
> > + dev_dbg(>pdev->dev, "Doorbell wait took %dms",
> > + jiffies_to_msecs(end) - jiffies_to_msecs(start));
> > + return 0;
> > +}
> > +
> > +static void cxl_mem_mbox_timeout(struct cxl_mem *cxlm,
> > +struct mbox_cmd *mbox_cmd)
> > +{
> > + dev_warn(>pdev->dev, "Mailbox command timed out\n");
> > + dev_info(>pdev->dev,
> > +"\topcode: 0x%04x\n"
> > +"\tpayload size: %zub\n",
> > +mbox_cmd->opcode, mbox_cmd->size_in);
> > +
> > + if (IS_ENABLED(CONFIG_CXL_MEM_INSECURE_DEBUG)) {
> > + print_hex_dump_debug("Payload ", DUMP_PREFIX_OFFSET, 16, 1,
> > +mbox_cmd->payload_in, mbox_cmd->size_in,
> > +true);
> > + }
> > +
> > + /* Here's a good place to figure out if a device reset is needed */
> > +}
> > +
> > +/**
> > + * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory device.
> > + * @cxlm: The CXL memory device to communicate with.
> > + * @mbox_cmd: Command to send to the memory device.
> > + *
> > + * Context: Any context. Expects mbox_lock to be held.
> > + * Return: -ETIMEDOUT if timeout occurred
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
> +#define cxl_doorbell_busy(cxlm)
> \
> + (cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET) &\
> + CXLDEV_MB_CTRL_DOORBELL)
> +
> +#define CXL_MAILBOX_TIMEOUT_US 2000
You been using the spec for the values. Is that number also from it ?
> +
> +enum opcode {
> + CXL_MBOX_OP_IDENTIFY= 0x4000,
> + CXL_MBOX_OP_MAX = 0x1
> +};
> +
> +/**
> + * struct mbox_cmd - A command to be submitted to hardware.
> + * @opcode: (input) The command set and command submitted to hardware.
> + * @payload_in: (input) Pointer to the input payload.
> + * @payload_out: (output) Pointer to the output payload. Must be allocated by
> + *the caller.
> + * @size_in: (input) Number of bytes to load from @payload.
> + * @size_out: (output) Number of bytes loaded into @payload.
> + * @return_code: (output) Error code returned from hardware.
> + *
> + * This is the primary mechanism used to send commands to the hardware.
> + * All the fields except @payload_* correspond exactly to the fields
> described in
> + * Command Register section of the CXL 2.0 spec (8.2.8.4.5). @payload_in and
> + * @payload_out are written to, and read from the Command Payload Registers
> + * defined in (8.2.8.4.8).
> + */
> +struct mbox_cmd {
> + u16 opcode;
> + void *payload_in;
> + void *payload_out;
On a 32-bit OS (not that we use those that more, but lets assume
someone really wants to), the void is 4-bytes, while on 64-bit it is
8-bytes.
`pahole` is your friend as I think there is a gap between opcode and
payload_in in the structure.
> + size_t size_in;
> + size_t size_out;
And those can also change depending on 32-bit/64-bit.
> + u16 return_code;
> +#define CXL_MBOX_SUCCESS 0
> +};
Do you want to use __packed to match with the spec?
Ah, reading later you don't care about it.
In that case may I recommend you move 'return_code' (or perhaps just
call it rc?) to be right after opcode? Less of gaps in that structure.
> +
> +static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> +{
> + const int timeout = msecs_to_jiffies(CXL_MAILBOX_TIMEOUT_US);
> + const unsigned long start = jiffies;
> + unsigned long end = start;
> +
> + while (cxl_doorbell_busy(cxlm)) {
> + end = jiffies;
> +
> + if (time_after(end, start + timeout)) {
> + /* Check again in case preempted before timeout test */
> + if (!cxl_doorbell_busy(cxlm))
> + break;
> + return -ETIMEDOUT;
> + }
> + cpu_relax();
> + }
Hm, that is not very scheduler friendly. I mean we are sitting here for
2000us (2 ms) - that is quite the amount of time spinning.
Should this perhaps be put in a workqueue?
> +
> + dev_dbg(>pdev->dev, "Doorbell wait took %dms",
> + jiffies_to_msecs(end) - jiffies_to_msecs(start));
> + return 0;
> +}
> +
> +static void cxl_mem_mbox_timeout(struct cxl_mem *cxlm,
> + struct mbox_cmd *mbox_cmd)
> +{
> + dev_warn(>pdev->dev, "Mailbox command timed out\n");
> + dev_info(>pdev->dev,
> + "\topcode: 0x%04x\n"
> + "\tpayload size: %zub\n",
> + mbox_cmd->opcode, mbox_cmd->size_in);
> +
> + if (IS_ENABLED(CONFIG_CXL_MEM_INSECURE_DEBUG)) {
> + print_hex_dump_debug("Payload ", DUMP_PREFIX_OFFSET, 16, 1,
> + mbox_cmd->payload_in, mbox_cmd->size_in,
> + true);
> + }
> +
> + /* Here's a good place to figure out if a device reset is needed */
> +}
> +
> +/**
> + * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory device.
> + * @cxlm: The CXL memory device to communicate with.
> + * @mbox_cmd: Command to send to the memory device.
> + *
> + * Context: Any context. Expects mbox_lock to be held.
> + * Return: -ETIMEDOUT if timeout occurred waiting for completion. 0 on
> success.
> + * Caller should check the return code in @mbox_cmd to make sure it
> + * succeeded.
> + *
> + * This is a generic form of the CXL mailbox send command, thus the only I/O
> + * operations used are cxl_read_mbox_reg(). Memory devices, and perhaps other
> + * types of CXL devices may have further information available upon error
> + * conditions.
> + *
> + * The CXL spec allows for up to two mailboxes. The intention is for the
> primary
> + * mailbox to be OS controlled and the secondary mailbox to be used by system
> + * firmware. This allows the OS and firmware to communicate with the device
> and
> + * not need to coordinate with each other. The driver only uses the primary
> + * mailbox.
> + */
> +static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
> + struct mbox_cmd *mbox_cmd)
> +{
> + void __iomem *payload = cxlm->mbox.regs + CXLDEV_MB_PAYLOAD_OFFSET;
> +
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On Fri, 29 Jan 2021, Ben Widawsky wrote:
> Provide enough functionality to utilize the mailbox of a memory device.
> The mailbox is used to interact with the firmware running on the memory
> device.
>
> The CXL specification defines separate capabilities for the mailbox and
> the memory device. The mailbox interface has a doorbell to indicate
> ready to accept commands and the memory device has a capability register
> that indicates the mailbox interface is ready. The expectation is that
> the doorbell-ready is always later than the memory-device-indication
> that the mailbox is ready.
>
> Create a function to handle sending a command, optionally with a
> payload, to the memory device, polling on a result, and then optionally
> copying out the payload. The algorithm for doing this comes straight out
> of the CXL 2.0 specification.
>
> Primary mailboxes are capable of generating an interrupt when submitting
> a command in the background. That implementation is saved for a later
> time.
>
> Secondary mailboxes aren't implemented at this time.
>
> The flow is proven with one implemented command, "identify". Because the
> class code has already told the driver this is a memory device and the
> identify command is mandatory.
>
> Signed-off-by: Ben Widawsky
> ---
> drivers/cxl/Kconfig | 14 ++
> drivers/cxl/cxl.h | 39 +
> drivers/cxl/mem.c | 342 +++-
> 3 files changed, 394 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
> index 3b66b46af8a0..fe591f74af96 100644
> --- a/drivers/cxl/Kconfig
> +++ b/drivers/cxl/Kconfig
> @@ -32,4 +32,18 @@ config CXL_MEM
> Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification.
>
> If unsure say 'm'.
> +
> +config CXL_MEM_INSECURE_DEBUG
> + bool "CXL.mem debugging"
> + depends on CXL_MEM
> + help
> + Enable debug of all CXL command payloads.
> +
> + Some CXL devices and controllers support encryption and other
> + security features. The payloads for the commands that enable
> + those features may contain sensitive clear-text security
> + material. Disable debug of those command payloads by default.
> + If you are a kernel developer actively working on CXL
> + security enabling say Y, otherwise say N.
Not specific to this patch, but the reference to encryption made me
curious about integrity: are all CXL.mem devices compatible with DIMP?
Some? None?
> +
> endif
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index a3da7f8050c4..df3d97154b63 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -31,9 +31,36 @@
> #define CXLDEV_MB_CAPS_OFFSET 0x00
> #define CXLDEV_MB_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0)
> #define CXLDEV_MB_CTRL_OFFSET 0x04
> +#define CXLDEV_MB_CTRL_DOORBELL BIT(0)
> #define CXLDEV_MB_CMD_OFFSET 0x08
> +#define CXLDEV_MB_CMD_COMMAND_OPCODE_MASK GENMASK(15, 0)
> +#define CXLDEV_MB_CMD_PAYLOAD_LENGTH_MASK GENMASK(36, 16)
> #define CXLDEV_MB_STATUS_OFFSET 0x10
> +#define CXLDEV_MB_STATUS_RET_CODE_MASK GENMASK(47, 32)
> #define CXLDEV_MB_BG_CMD_STATUS_OFFSET 0x18
> +#define CXLDEV_MB_PAYLOAD_OFFSET 0x20
> +
> +/* Memory Device (CXL 2.0 - 8.2.8.5.1.1) */
> +#define CXLMDEV_STATUS_OFFSET 0x0
> +#define CXLMDEV_DEV_FATAL BIT(0)
> +#define CXLMDEV_FW_HALT BIT(1)
> +#define CXLMDEV_STATUS_MEDIA_STATUS_MASK GENMASK(3, 2)
> +#define CXLMDEV_MS_NOT_READY 0
> +#define CXLMDEV_MS_READY 1
> +#define CXLMDEV_MS_ERROR 2
> +#define CXLMDEV_MS_DISABLED 3
> +#define CXLMDEV_READY(status) \
> + (CXL_GET_FIELD(status, CXLMDEV_STATUS_MEDIA_STATUS) ==
> CXLMDEV_MS_READY)
> +#define CXLMDEV_MBOX_IF_READY BIT(4)
> +#define CXLMDEV_RESET_NEEDED_SHIFT 5
> +#define CXLMDEV_RESET_NEEDED_MASK GENMASK(7, 5)
> +#define CXLMDEV_RESET_NEEDED_NOT 0
> +#define CXLMDEV_RESET_NEEDED_COLD 1
> +#define CXLMDEV_RESET_NEEDED_WARM 2
> +#define CXLMDEV_RESET_NEEDED_HOT 3
> +#define CXLMDEV_RESET_NEEDED_CXL 4
> +#define CXLMDEV_RESET_NEEDED(status) \
> + (CXL_GET_FIELD(status, CXLMDEV_RESET_NEEDED) !=
> CXLMDEV_RESET_NEEDED_NOT)
>
> /**
> * struct cxl_mem - A CXL memory device
> @@ -44,6 +71,16 @@ struct cxl_mem {
> struct pci_dev *pdev;
> void __iomem *regs;
>
> + struct {
> + struct range range;
> + } pmem;
> +
> + struct {
> + struct range range;
> + } ram;
> +
> + char firmware_version[0x10];
> +
> /* Cap 0001h - CXL_CAP_CAP_ID_DEVICE_STATUS */
> struct {
> void __iomem *regs;
> @@ -51,6 +88,7 @@ struct cxl_mem {
>
> /* Cap 0002h - CXL_CAP_CAP_ID_PRIMARY_MAILBOX */
> struct {
> + struct mutex mutex; /* Protects device mailbox and firmware */
> void __iomem *regs;
> size_t payload_size;
> } mbox;
> @@ -89,5 +127,6 @@ struct cxl_mem {
>
> cxl_reg(status);
> cxl_reg(mbox);
