Re: [ndctl PATCH v2 07/13] test: introduce a libcxl unit test
On 21-02-18 19:03:25, Vishal Verma wrote:
> Add a new 'libcxl' test containing a basic harness for unit testing
> libcxl APIs. Include sanity tests such as making sure the test is
> running in an emulated environment, the ability to load and unload
> modules. Submit an 'Identify Device' command, and verify that it
> succeeds, and the identify data returned is as expected from an emulated
> QEMU device.
>
> Cc: Ben Widawsky
> Cc: Dan Williams
> Signed-off-by: Vishal Verma
> ---
> test/libcxl-expect.h | 13 +++
> test/libcxl.c| 268 +++
> test/Makefile.am | 12 +-
> 3 files changed, 291 insertions(+), 2 deletions(-)
> create mode 100644 test/libcxl-expect.h
> create mode 100644 test/libcxl.c
>
> diff --git a/test/libcxl-expect.h b/test/libcxl-expect.h
> new file mode 100644
> index 000..acb8db9
> --- /dev/null
> +++ b/test/libcxl-expect.h
> @@ -0,0 +1,13 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright (C) 2021 Intel Corporation. All rights reserved. */
> +#ifndef __LIBCXL_EXPECT_H__
> +#define __LIBCXL_EXPECT_H__
> +#include
> +
> +#define EXPECT_FW_VER "BWFW VERSION 00"
> +
> +/* Identify command fields */
> +#define EXPECT_CMD_IDENTIFY_PARTITION_ALIGN 0ULL
> +#define EXPECT_CMD_IDENTIFY_LSA_SIZE 1024U
> +
> +#endif /* __LIBCXL_EXPECT_H__ */
> diff --git a/test/libcxl.c b/test/libcxl.c
> new file mode 100644
> index 000..9662f34
> --- /dev/null
> +++ b/test/libcxl.c
> @@ -0,0 +1,268 @@
> +// SPDX-License-Identifier: LGPL-2.1
> +/* Copyright (C) 2021, Intel Corporation. All rights reserved. */
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include "libcxl-expect.h"
> +
> +#define TEST_SKIP 77
> +
> +const char *mod_list[] = {
> + "cxl_mem",
> + "cxl_bus",
> +};
> +
> +static int test_cxl_presence(struct cxl_ctx *ctx)
> +{
> + struct cxl_memdev *memdev;
> + int count = 0;
> +
> + cxl_memdev_foreach(ctx, memdev)
> + count++;
> +
> + if (count == 0) {
> + fprintf(stderr, "%s: no cxl memdevs found\n", __func__);
> + return TEST_SKIP;
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * Only continue with tests if all CXL devices in the system are
> qemu-emulated
> + * 'fake' devices. For now, use the firmware_version to check for this.
> Later,
> + * this might need to be changed to a vendor specific command.
> + *
I'd like to have "Space" to potentially differentiate firmware. How about for
now chop off the "version" from the comparison.
`strncmp(fw, EXPECT_FW_VER, 14)`
Please add a comment why it's not all 16 bytes so I don't ask you later :D
I don't know of a great way to do this other than vendor specific command.
> + * TODO: Change this to produce a list of devices that are safe to run tests
> + * against, and only run subsequent tests on this list. That will allow
> devices
> + * from other, non-emulated sources to be present in the system, and still
> run
> + * these unit tests safely.
> + */
> +static int test_cxl_emulation_env(struct cxl_ctx *ctx)
> +{
> + struct cxl_memdev *memdev;
> +
> + cxl_memdev_foreach(ctx, memdev) {
> + const char *fw;
> +
> + fw = cxl_memdev_get_firmware_verison(memdev);
> + if (!fw)
> + return -ENXIO;
> + if (strcmp(fw, EXPECT_FW_VER) != 0) {
> + fprintf(stderr,
> + "%s: found non-emulation device, aborting\n",
> + __func__);
> + return TEST_SKIP;
> + }
> + }
> + return 0;
> +}
> +
> +static int test_cxl_modules(struct cxl_ctx *ctx)
> +{
> + int rc;
> + unsigned int i;
> + const char *name;
> + struct kmod_module *mod;
> + struct kmod_ctx *kmod_ctx;
> +
> + kmod_ctx = kmod_new(NULL, NULL);
> + if (!kmod_ctx)
> + return -ENXIO;
> + kmod_set_log_priority(kmod_ctx, LOG_DEBUG);
> +
> + /* test module removal */
> + for (i = 0; i < ARRAY_SIZE(mod_list); i++) {
> + int state;
> +
> + name
Re: [ndctl PATCH v2 04/13] libcxl: add support for the 'Identify Device' command
On 21-02-18 19:03:22, Vishal Verma wrote:
> Add APIs to allocate and send an 'Identify Device' command, and
> accessors to retrieve some of the fields from the resulting data.
>
> Only add a handful accessor functions; more can be added as the need
> arises. The fields added are fw_revision, partition_align, and
> lsa_size.
>
How do we feel about for each command the ability to return n bytes as an opaque
object? That way, if some client application wants more of the identify data
they can get it without needing library API to do it.
/**
* cxl_cmd_identify_get_payload() - Return output payload
* @cmd: Identify command.
* @n: Max number of bytes to return.
* @out: Buffer to copy to.
*
* Returns actual number of bytes copied out.
*
* Returns up to n bytes in (CXL native format) of data from the output payload
* of the identify command.
*/
CXL_EXPORT unsigned int
cxl_cmd_identify_get_payload(struct cxl_cmd *cmd, unsigned int n, void *out)
{
}
I recall discussing this briefly, but I don't recall what we had decided.
> Cc: Ben Widawsky
> Cc: Dan Williams
> Signed-off-by: Vishal Verma
> ---
> cxl/lib/private.h | 19 ++
> cxl/lib/libcxl.c | 49 ++
> cxl/libcxl.h | 4
> cxl/lib/libcxl.sym | 4
> 4 files changed, 76 insertions(+)
>
> diff --git a/cxl/lib/private.h b/cxl/lib/private.h
> index 87ca17e..3273f21 100644
> --- a/cxl/lib/private.h
> +++ b/cxl/lib/private.h
> @@ -54,6 +54,25 @@ struct cxl_cmd {
> int status;
> };
>
> +#define CXL_CMD_IDENTIFY_FW_REV_LENGTH 0x10
> +
> +struct cxl_cmd_identify {
> + char fw_revision[CXL_CMD_IDENTIFY_FW_REV_LENGTH];
> + le64 total_capacity;
> + le64 volatile_capacity;
> + le64 persistent_capacity;
> + le64 partition_align;
> + le16 info_event_log_size;
> + le16 warning_event_log_size;
> + le16 failure_event_log_size;
> + le16 fatal_event_log_size;
> + le32 lsa_size;
> + u8 poison_list_max_mer[3];
> + le16 inject_poison_limit;
> + u8 poison_caps;
> + u8 qos_telemetry_caps;
> +} __attribute__((packed));
> +
> static inline int check_kmod(struct kmod_ctx *kmod_ctx)
> {
> return kmod_ctx ? 0 : -ENXIO;
> diff --git a/cxl/lib/libcxl.c b/cxl/lib/libcxl.c
> index f1155a1..4751cba 100644
> --- a/cxl/lib/libcxl.c
> +++ b/cxl/lib/libcxl.c
> @@ -13,7 +13,10 @@
> #include
> #include
> #include
> +#include
> +#include
> #include
> +#include
>
> #include
> #include
> @@ -675,6 +678,52 @@ CXL_EXPORT const char *cxl_cmd_get_devname(struct
> cxl_cmd *cmd)
> return cxl_memdev_get_devname(cmd->memdev);
> }
>
> +CXL_EXPORT struct cxl_cmd *cxl_cmd_new_identify(struct cxl_memdev *memdev)
> +{
> + return cxl_cmd_new_generic(memdev, CXL_MEM_COMMAND_ID_IDENTIFY);
> +}
> +
> +CXL_EXPORT int cxl_cmd_identify_get_fw_rev(struct cxl_cmd *cmd, char *fw_rev,
> + int fw_len)
> +{
> + struct cxl_cmd_identify *id = (void *)cmd->send_cmd->out.payload;
> +
> + if (cmd->send_cmd->id != CXL_MEM_COMMAND_ID_IDENTIFY)
> + return -EINVAL;
> + if (cmd->status < 0)
> + return cmd->status;
> +
> + if (fw_len > 0)
> + memcpy(fw_rev, id->fw_revision,
> + min(fw_len, CXL_CMD_IDENTIFY_FW_REV_LENGTH));
> + return 0;
> +}
> +
> +CXL_EXPORT unsigned long long cxl_cmd_identify_get_partition_align(
> + struct cxl_cmd *cmd)
> +{
> + struct cxl_cmd_identify *id = (void *)cmd->send_cmd->out.payload;
> +
> + if (cmd->send_cmd->id != CXL_MEM_COMMAND_ID_IDENTIFY)
> + return -EINVAL;
> + if (cmd->status < 0)
> + return cmd->status;
> +
> + return le64_to_cpu(id->partition_align);
> +}
> +
> +CXL_EXPORT unsigned int cxl_cmd_identify_get_lsa_size(struct cxl_cmd *cmd)
> +{
> + struct cxl_cmd_identify *id = (void *)cmd->send_cmd->out.payload;
> +
> + if (cmd->send_cmd->id != CXL_MEM_COMMAND_ID_IDENTIFY)
> + return -EINVAL;
> + if (cmd->status < 0)
> + return cmd->status;
> +
> + return le32_to_cpu(id->lsa_size);
> +}
> +
> CXL_EXPORT struct cxl_cmd *cxl_cmd_new_raw(struct cxl_memdev *memdev,
> int opcode)
> {
> diff --git a/cxl/libcxl.h b/cxl/libcxl.h
> index 6e87b80..9ed8c83 100644
> --- a/cxl/libcxl.h
> +++ b/cxl/libcxl.h
> @@ -58,6 +58,10 @@ void cxl_cmd_unref(struct cxl_cmd *cmd);
> int cxl_cmd_submit(struct cxl_cmd *cmd);
> int cxl_cmd_get_mbox_status(struct cxl_cmd *cmd);
> int
Re: [ndctl PATCH v2 03/13] libcxl: add support for command query and submission
On 21-02-18 19:03:21, Vishal Verma wrote:
> Add a set of APIs around 'cxl_cmd' for querying the kernel for supported
> commands, allocating and validating command structures against the
> supported set, and submitting the commands.
>
> 'Query Commands' and 'Send Command' are implemented as IOCTLs in the
> kernel. 'Query Commands' returns information about each supported
> command, such as flags governing its use, or input and output payload
> sizes. This information is used to validate command support, as well as
> set up input and output buffers for command submission.
>
> Cc: Ben Widawsky
> Cc: Dan Williams
> Signed-off-by: Vishal Verma
> ---
> cxl/lib/private.h | 33
> cxl/lib/libcxl.c | 393 +
> cxl/libcxl.h | 11 ++
> cxl/lib/libcxl.sym | 13 ++
> 4 files changed, 450 insertions(+)
>
> diff --git a/cxl/lib/private.h b/cxl/lib/private.h
> index fc88fa1..87ca17e 100644
> --- a/cxl/lib/private.h
> +++ b/cxl/lib/private.h
> @@ -4,6 +4,9 @@
> #define _LIBCXL_PRIVATE_H_
>
> #include
> +#include
> +#include
> +#include
>
> #define CXL_EXPORT __attribute__ ((visibility("default")))
>
> @@ -21,6 +24,36 @@ struct cxl_memdev {
> struct kmod_module *module;
> };
>
> +enum cxl_cmd_query_status {
> + CXL_CMD_QUERY_NOT_RUN = 0,
> + CXL_CMD_QUERY_OK,
> + CXL_CMD_QUERY_UNSUPPORTED,
> +};
> +
> +/**
> + * struct cxl_cmd - CXL memdev command
> + * @memdev: the memory device to which the command is being sent
> + * @query_cmd: structure for the Linux 'Query commands' ioctl
> + * @send_cmd: structure for the Linux 'Send command' ioctl
> + * @input_payload: buffer for input payload managed by libcxl
> + * @output_payload: buffer for output payload managed by libcxl
> + * @refcount: reference for passing command buffer around
> + * @query_status: status from query_commands
> + * @query_idx: index of 'this' command in the query_commands array
> + * @status: command return status from the device
> + */
> +struct cxl_cmd {
> + struct cxl_memdev *memdev;
> + struct cxl_mem_query_commands *query_cmd;
> + struct cxl_send_command *send_cmd;
> + void *input_payload;
> + void *output_payload;
> + int refcount;
> + int query_status;
> + int query_idx;
> + int status;
> +};
> +
> static inline int check_kmod(struct kmod_ctx *kmod_ctx)
> {
> return kmod_ctx ? 0 : -ENXIO;
> diff --git a/cxl/lib/libcxl.c b/cxl/lib/libcxl.c
> index d34e7d0..f1155a1 100644
> --- a/cxl/lib/libcxl.c
> +++ b/cxl/lib/libcxl.c
> @@ -9,6 +9,7 @@
> #include
> #include
> #include
> +#include
> #include
> #include
> #include
> @@ -17,6 +18,7 @@
> #include
> #include
> #include
> +#include
> #include
> #include "private.h"
>
> @@ -343,3 +345,394 @@ CXL_EXPORT const char
> *cxl_memdev_get_firmware_verison(struct cxl_memdev *memdev
> {
> return memdev->firmware_version;
> }
> +
> +CXL_EXPORT void cxl_cmd_unref(struct cxl_cmd *cmd)
> +{
> + if (!cmd)
> + return;
> + if (--cmd->refcount == 0) {
> + free(cmd->query_cmd);
> + free(cmd->send_cmd);
> + free(cmd->input_payload);
> + free(cmd->output_payload);
> + free(cmd);
> + }
> +}
> +
> +CXL_EXPORT void cxl_cmd_ref(struct cxl_cmd *cmd)
> +{
> + cmd->refcount++;
> +}
> +
> +static int cxl_cmd_alloc_query(struct cxl_cmd *cmd, int num_cmds)
> +{
> + size_t size;
> +
> + if (!cmd)
> + return -EINVAL;
> +
> + if (cmd->query_cmd != NULL)
> + free(cmd->query_cmd);
> +
> + size = sizeof(struct cxl_mem_query_commands) +
> + (num_cmds * sizeof(struct cxl_command_info));
> + cmd->query_cmd = calloc(1, size);
> + if (!cmd->query_cmd)
> + return -ENOMEM;
> +
> + cmd->query_cmd->n_commands = num_cmds;
> +
> + return 0;
> +}
> +
> +static struct cxl_cmd *cxl_cmd_new(struct cxl_memdev *memdev)
> +{
> + struct cxl_cmd *cmd;
> + size_t size;
> +
> + size = sizeof(*cmd);
> + cmd = calloc(1, size);
> + if (!cmd)
> + return NULL;
> +
> + cxl_cmd_ref(cmd);
> + cmd->memdev = memdev;
> +
> + return cmd;
> +}
> +
> +static int __do_cmd(struct cxl_cmd *cmd, int ioctl_cmd, int fd)
> +{
> + void *cmd_buf;
> + int rc;
> +
> + switch (ioctl_cmd) {
>
Re: [ndctl PATCH v2 02/13] cxl: add a local copy of the cxl_mem UAPI header
On 21-02-18 19:03:20, Vishal Verma wrote:
> While CXL functionality is under development, it is useful to have a
> local copy of the UAPI header for cxl_mem definitions. This allows
> building cxl and libcxl on systems where the appropriate kernel headers
> are not installed in the usual locations.
It it meant to be able to use a system, or dev copy of cxl_mem.h? Could you
maybe spell out in the commit message how one could select between the two?
>
> Cc: Ben Widawsky
> Cc: Dan Williams
> Signed-off-by: Vishal Verma
> ---
> Makefile.am | 3 +-
> Makefile.am.in | 1 +
> cxl/cxl_mem.h | 181
> cxl/lib/Makefile.am | 2 +-
> 4 files changed, 185 insertions(+), 2 deletions(-)
> create mode 100644 cxl/cxl_mem.h
>
> diff --git a/Makefile.am b/Makefile.am
> index 428fd40..4904ee7 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -89,4 +89,5 @@ libutil_a_SOURCES = \
>
> nobase_include_HEADERS = \
> daxctl/libdaxctl.h \
> - cxl/libcxl.h
> + cxl/libcxl.h \
> + cxl/cxl_mem.h
> diff --git a/Makefile.am.in b/Makefile.am.in
> index aaeee53..a748128 100644
> --- a/Makefile.am.in
> +++ b/Makefile.am.in
> @@ -11,6 +11,7 @@ AM_CPPFLAGS = \
> -DNDCTL_MAN_PATH=\""$(mandir)"\" \
> -I${top_srcdir}/ndctl/lib \
> -I${top_srcdir}/ndctl \
> + -I${top_srcdir}/cxl \
> -I${top_srcdir}/ \
> $(KMOD_CFLAGS) \
> $(UDEV_CFLAGS) \
> diff --git a/cxl/cxl_mem.h b/cxl/cxl_mem.h
> new file mode 100644
> index 000..be9c7ad
> --- /dev/null
> +++ b/cxl/cxl_mem.h
> @@ -0,0 +1,181 @@
> +/* SPDX-License-Identifier: LGPL-2.1 */
> +/* Copyright (C) 2020-2021, Intel Corporation. All rights reserved. */
> +/*
> + * CXL IOCTLs for Memory Devices
> + */
> +
> +#ifndef _UAPI_CXL_MEM_H_
> +#define _UAPI_CXL_MEM_H_
> +
> +#include
> +#include
> +#include
> +
> +#define __user
> +
> +/**
> + * DOC: UAPI
> + *
> + * CXL memory devices expose UAPI to have a standard user interface.
> + * Userspace can refer to these structure definitions and UAPI formats
> + * to communicate to driver. The commands themselves are somewhat obfuscated
> + * with macro magic. They have the form CXL_MEM_COMMAND_ID_.
> + *
> + * For example "CXL_MEM_COMMAND_ID_INVALID"
> + *
> + * Not all of all commands that the driver supports are always available for
> use
> + * by userspace. Userspace must check the results from the QUERY command in
> + * order to determine the live set of commands.
> + */
> +
> +#define CXL_MEM_QUERY_COMMANDS _IOR(0xCE, 1, struct cxl_mem_query_commands)
> +#define CXL_MEM_SEND_COMMAND _IOWR(0xCE, 2, struct cxl_send_command)
> +
> +#define CXL_CMDS \
> + ___C(INVALID, "Invalid Command"), \
> + ___C(IDENTIFY, "Identify Command"), \
> + ___C(RAW, "Raw device command"), \
> + ___C(GET_SUPPORTED_LOGS, "Get Supported Logs"), \
> + ___C(GET_FW_INFO, "Get FW Info"), \
> + ___C(GET_PARTITION_INFO, "Get Partition Information"),\
> + ___C(GET_LSA, "Get Label Storage Area"), \
> + ___C(GET_HEALTH_INFO, "Get Health Info"), \
> + ___C(GET_LOG, "Get Log"), \
> + ___C(MAX, "Last command")
> +
> +#define ___C(a, b) CXL_MEM_COMMAND_ID_##a
> +enum { CXL_CMDS };
> +
> +#undef ___C
> +#define ___C(a, b) { b }
> +static const struct {
> + const char *name;
> +} cxl_command_names[] = { CXL_CMDS };
> +#undef ___C
> +
> +/**
> + * struct cxl_command_info - Command information returned from a query.
> + * @id: ID number for the command.
> + * @flags: Flags that specify command behavior.
> + *
> + * * %CXL_MEM_COMMAND_FLAG_KERNEL: This command is reserved for exclusive
> + *kernel use.
> + * * %CXL_MEM_COMMAND_FLAG_MUTEX: This command may require coordination with
> + *the kernel in order to complete successfully.
> + *
> + * @size_in: Expected input size, or -1 if variable length.
> + * @size_out: Expected output size, or -1 if variable length.
> + *
> + * Represents a single command that is supported by both the driver and the
> + * hardware. This is returned as part of an array from the query ioctl. The
> + * following would be a command named "foobar" that takes a variable length
> + * input and returns
Re: [ndctl PATCH v2 01/13] cxl: add a cxl utility and libcxl library
On 21-02-18 19:03:19, Vishal Verma wrote:
> CXL - or Compute eXpress Link - is a new interconnect that extends PCIe
> to support a wide range of devices, including cache coherent memory
> expanders. As such, these devices can be new sources of 'persistent
> memory', and the 'ndctl' umbrella of tools and libraries needs to be able
> to interact with them.
>
> Add a new utility and library for managing these CXL memory devices. This
> is an initial bring-up for interacting with CXL devices, and only includes
> adding the utility and library infrastructure, parsing device information
> from sysfs for CXL devices, and providing a 'cxl-list' command to
> display this information in JSON formatted output.
>
> Cc: Ben Widawsky
> Cc: Dan Williams
> Signed-off-by: Vishal Verma
A couple of really minor things below
> ---
> Documentation/cxl/cxl-list.txt | 65 +
> Documentation/cxl/cxl.txt| 34 +++
> Documentation/cxl/human-option.txt | 8 +
> Documentation/cxl/verbose-option.txt | 5 +
> configure.ac | 3 +
> Makefile.am | 8 +-
> Makefile.am.in | 4 +
> cxl/lib/private.h| 29 +++
> cxl/lib/libcxl.c | 345 +++
> cxl/builtin.h| 8 +
> cxl/libcxl.h | 55 +
> util/filter.h| 2 +
> util/json.h | 3 +
> util/main.h | 3 +
> cxl/cxl.c| 95
> cxl/list.c | 113 +
> util/filter.c| 20 ++
> util/json.c | 26 ++
> .gitignore | 2 +
> Documentation/cxl/Makefile.am| 58 +
> cxl/Makefile.am | 21 ++
> cxl/lib/Makefile.am | 32 +++
> cxl/lib/libcxl.pc.in | 11 +
> cxl/lib/libcxl.sym | 29 +++
> 24 files changed, 976 insertions(+), 3 deletions(-)
> create mode 100644 Documentation/cxl/cxl-list.txt
> create mode 100644 Documentation/cxl/cxl.txt
> create mode 100644 Documentation/cxl/human-option.txt
> create mode 100644 Documentation/cxl/verbose-option.txt
> create mode 100644 cxl/lib/private.h
> create mode 100644 cxl/lib/libcxl.c
> create mode 100644 cxl/builtin.h
> create mode 100644 cxl/libcxl.h
> create mode 100644 cxl/cxl.c
> create mode 100644 cxl/list.c
> create mode 100644 Documentation/cxl/Makefile.am
> create mode 100644 cxl/Makefile.am
> create mode 100644 cxl/lib/Makefile.am
> create mode 100644 cxl/lib/libcxl.pc.in
> create mode 100644 cxl/lib/libcxl.sym
>
> diff --git a/Documentation/cxl/cxl-list.txt b/Documentation/cxl/cxl-list.txt
> new file mode 100644
> index 000..107b388
> --- /dev/null
> +++ b/Documentation/cxl/cxl-list.txt
> @@ -0,0 +1,65 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +cxl-list(1)
> +===
> +
> +NAME
> +
> +cxl-list - CXL capable host bridges, switches, devices, and their attributes
> +in json.
> +
> +SYNOPSIS
> +
> +[verse]
> +'cxl list' []
> +
> +Walk the CXL capable device hierarchy in the system and list all device
> +instances along with some of their major attributes.
This doesn't seem to match the above. Here it's just devices and above you talk
about bridges and switches as well.
> +
> +Options can be specified to limit the output to specific devices.
> +By default, 'cxl list' with no options is equivalent to:
> +[verse]
> +cxl list --devices
> +
> +EXAMPLE
> +---
> +
> +# cxl list --devices
> +{
> + "memdev":"mem0",
> + "pmem_size":268435456,
> + "ram_size":0,
> +}
> +
> +
> +OPTIONS
> +---
> +-d::
> +--dev=::
> + Specify a cxl device name to filter the listing. For example:
> +
> +# cxl list --dev=mem0
> +{
> + "memdev":"mem0",
> + "pmem_size":268435456,
> + "ram_size":0,
> +}
> +
> +
> +-D::
> +--devices::
> + Include all CXL devices in the listing
> +
> +-i::
> +--idle::
> + Include idle (not enabled / zero-sized) devices in the listing
> +
> +include::human-option.txt[]
> +
> +include::verbose-option.txt[]
> +
> +include::../copyright.txt[]
> +
> +SEE ALSO
> +
> +linkcxl:ndctl-list[1]
> diff --git a/Documentation/cxl/cxl.txt b/Documentation/cxl/cxl.txt
> new file mode 100644
> index 000..e99e61b
> --- /dev/null
[PATCH v2] cxl/mem: Fix potential memory leak
When submitting a command for userspace, input and output payload bounce
buffers are allocated. For a given command, both input and output
buffers may exist and so when allocation of the input buffer fails, the
output buffer must be freed too.
As far as I can tell, userspace can't easily exploit the leak to OOM a
machine unless the machine was already near OOM state.
Fixes: 583fa5e71cae ("cxl/mem: Add basic IOCTL interface")
Reported-by: Konrad Rzeszutek Wilk
Signed-off-by: Ben Widawsky
---
drivers/cxl/mem.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index df895bcca63a..244cb7d89678 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -514,8 +514,10 @@ static int handle_mailbox_cmd_from_user(struct cxl_mem
*cxlm,
if (cmd->info.size_in) {
mbox_cmd.payload_in = vmemdup_user(u64_to_user_ptr(in_payload),
cmd->info.size_in);
- if (IS_ERR(mbox_cmd.payload_in))
+ if (IS_ERR(mbox_cmd.payload_in)) {
+ kvfree(mbox_cmd.payload_out);
return PTR_ERR(mbox_cmd.payload_in);
+ }
}
rc = cxl_mem_mbox_get(cxlm);
--
2.30.1
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH] cxl/mem: Fixes to IOCTL interface
On 21-02-20 18:38:36, Dan Williams wrote:
> On Sat, Feb 20, 2021 at 1:57 PM Ben Widawsky wrote:
> >
> > When submitting a command for userspace, input and output payload bounce
> > buffers are allocated. For a given command, both input and output
> > buffers may exist and so when allocation of the input buffer fails, the
> > output buffer must be freed. As far as I can tell, userspace can't
> > easily exploit the leak to OOM a machine unless the machine was already
> > near OOM state.
> >
> > This bug was introduced in v5 of the patch and did not exist in prior
> > revisions.
> >
>
> Thanks for the quick turnaround, but I think that speed introduced
> some issues...
>
> > While here, adjust the variable 'j' found in patch review by Konrad.
>
> Please split this pure cleanup to its own patch. The subject says
> "Fixes", but it's only the one fix.
>
This was intentional. I pinged you internally to just drop it if you don't like
to combine these kind of things. It didn't feel worthwhile to introduce a new
patch to change the 'j'. I agree with Konrad that 'j' is not the best variable
name to use. Konrad, maybe you'd like to send a fixup for that one?
I will drop this hunk.
> >
> > Cc: Al Viro
> > Reported-by: Konrad Rzeszutek Wilk
>
> Since the commit is upstream add a "Fixes" line:
>
> Fixes: 583fa5e71cae ('cxl/mem: Add basic IOCTL interface")
>
> > Signed-off-by: Ben Widawsky
> > Reviewed-by: Dan Williams (v2)
> > Reviewed-by: Jonathan Cameron
>
> Jonathan and I didn't pre-review this.
My bad on this. It was a mistake that I pulled the info from the original patch
I was fixing.
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[PATCH] cxl/mem: Fixes to IOCTL interface
When submitting a command for userspace, input and output payload bounce
buffers are allocated. For a given command, both input and output
buffers may exist and so when allocation of the input buffer fails, the
output buffer must be freed. As far as I can tell, userspace can't
easily exploit the leak to OOM a machine unless the machine was already
near OOM state.
This bug was introduced in v5 of the patch and did not exist in prior
revisions.
While here, adjust the variable 'j' found in patch review by Konrad.
Cc: Al Viro
Reported-by: Konrad Rzeszutek Wilk
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams (v2)
Reviewed-by: Jonathan Cameron
---
drivers/cxl/mem.c | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index df895bcca63a..626fd7066f4f 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -514,8 +514,10 @@ static int handle_mailbox_cmd_from_user(struct cxl_mem
*cxlm,
if (cmd->info.size_in) {
mbox_cmd.payload_in = vmemdup_user(u64_to_user_ptr(in_payload),
cmd->info.size_in);
- if (IS_ERR(mbox_cmd.payload_in))
+ if (IS_ERR(mbox_cmd.payload_in)) {
+ kvfree(mbox_cmd.payload_out);
return PTR_ERR(mbox_cmd.payload_in);
+ }
}
rc = cxl_mem_mbox_get(cxlm);
@@ -696,7 +698,7 @@ static int cxl_query_cmd(struct cxl_memdev *cxlmd,
struct device *dev = &cxlmd->dev;
struct cxl_mem_command *cmd;
u32 n_commands;
- int j = 0;
+ int cmds = 0;
dev_dbg(dev, "Query IOCTL\n");
@@ -714,10 +716,10 @@ static int cxl_query_cmd(struct cxl_memdev *cxlmd,
cxl_for_each_cmd(cmd) {
const struct cxl_command_info *info = &cmd->info;
- if (copy_to_user(&q->commands[j++], info, sizeof(*info)))
+ if (copy_to_user(&q->commands[cmds++], info, sizeof(*info)))
return -EFAULT;
- if (j == n_commands)
+ if (cmds == n_commands)
break;
}
--
2.30.1
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH v5 4/9] cxl/mem: Add basic IOCTL interface
On 21-02-19 20:22:00, Konrad Rzeszutek Wilk wrote:
> ..snip..
> > +static int handle_mailbox_cmd_from_user(struct cxl_mem *cxlm,
> > + const struct cxl_mem_command *cmd,
> > + u64 in_payload, u64 out_payload,
> > + s32 *size_out, u32 *retval)
> > +{
> > + struct device *dev = &cxlm->pdev->dev;
> > + struct mbox_cmd mbox_cmd = {
> > + .opcode = cmd->opcode,
> > + .size_in = cmd->info.size_in,
> > + .size_out = cmd->info.size_out,
> > + };
> > + int rc;
> > +
> > + if (cmd->info.size_out) {
> > + mbox_cmd.payload_out = kvzalloc(cmd->info.size_out, GFP_KERNEL);
> > + if (!mbox_cmd.payload_out)
> > + return -ENOMEM;
> > + }
> > +
> > + if (cmd->info.size_in) {
> > + mbox_cmd.payload_in = vmemdup_user(u64_to_user_ptr(in_payload),
> > + cmd->info.size_in);
> > + if (IS_ERR(mbox_cmd.payload_in))
> > + return PTR_ERR(mbox_cmd.payload_in);
>
> Not that this should happen, but what if info.size_out was set? Should
> you also free mbox_cmd.payload_out?
>
Thanks Konrad.
Dan, do you want me to send a fixup patch? This bug was introduced from v4->v5.
> > + }
> > +
> > + rc = cxl_mem_mbox_get(cxlm);
> > + if (rc)
> > + goto out;
> > +
> > + dev_dbg(dev,
> > + "Submitting %s command for user\n"
> > + "\topcode: %x\n"
> > + "\tsize: %ub\n",
> > + cxl_command_names[cmd->info.id].name, mbox_cmd.opcode,
> > + cmd->info.size_in);
> > +
> > + rc = __cxl_mem_mbox_send_cmd(cxlm, &mbox_cmd);
> > + cxl_mem_mbox_put(cxlm);
> > + if (rc)
> > + goto out;
> > +
> > + /*
> > +* @size_out contains the max size that's allowed to be written back out
> > +* to userspace. While the payload may have written more output than
> > +* this it will have to be ignored.
> > +*/
> > + if (mbox_cmd.size_out) {
> > + dev_WARN_ONCE(dev, mbox_cmd.size_out > *size_out,
> > + "Invalid return size\n");
> > + if (copy_to_user(u64_to_user_ptr(out_payload),
> > +mbox_cmd.payload_out, mbox_cmd.size_out)) {
> > + rc = -EFAULT;
> > + goto out;
> > + }
> > + }
> > +
> > + *size_out = mbox_cmd.size_out;
> > + *retval = mbox_cmd.return_code;
> > +
> > +out:
> > + kvfree(mbox_cmd.payload_in);
> > + kvfree(mbox_cmd.payload_out);
> > + return rc;
> > +}
>
> ..snip..
>
> > +static int cxl_query_cmd(struct cxl_memdev *cxlmd,
> > +struct cxl_mem_query_commands __user *q)
> > +{
> > + struct device *dev = &cxlmd->dev;
> > + struct cxl_mem_command *cmd;
> > + u32 n_commands;
> > + int j = 0;
>
> How come it is 'j' instead of the usual 'i'?
Just how it got split out/copied from an earlier version of the series.
I think rename to i, or cmds would be best as well. I/Dan can do it as part of
the bug fix you found above.
> > +
> > + dev_dbg(dev, "Query IOCTL\n");
> > +
> > + if (get_user(n_commands, &q->n_commands))
> > + return -EFAULT;
> > +
> > + /* returns the total number if 0 elements are requested. */
> > + if (n_commands == 0)
> > + return put_user(cxl_cmd_count, &q->n_commands);
> > +
> > + /*
> > +* otherwise, return max(n_commands, total commands) cxl_command_info
> > +* structures.
> > +*/
> > + cxl_for_each_cmd(cmd) {
> > + const struct cxl_command_info *info = &cmd->info;
> > +
> > + if (copy_to_user(&q->commands[j++], info, sizeof(*info)))
> > + return -EFAULT;
> > +
> > + if (j == n_commands)
> > + break;
> > + }
> > +
> > + return 0;
> > +}
> > +
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[RFC PATCH v5 9/9] cxl/mem: Add payload dumping for debug
It's often useful in debug scenarios to see what the hardware has dumped
out. As it stands today, any device error will result in the payload not
being copied out, so there is no way to triage commands which weren't
expected to fail (and sometimes the payload may have that information).
The functionality is protected by normal kernel security mechanisms as
well as a CONFIG option in the CXL driver.
This was extracted from the original version of the CXL enabling patch
series.
Signed-off-by: Ben Widawsky
---
drivers/cxl/Kconfig | 13 +
drivers/cxl/mem.c | 8
2 files changed, 21 insertions(+)
diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
index 97dc4d751651..3eec9276e586 100644
--- a/drivers/cxl/Kconfig
+++ b/drivers/cxl/Kconfig
@@ -50,4 +50,17 @@ config CXL_MEM_RAW_COMMANDS
potential impact to memory currently in use by the kernel.
If developing CXL hardware or the driver say Y, otherwise say N.
+
+config CXL_MEM_INSECURE_DEBUG
+ bool "CXL.mem debugging"
+ depends on CXL_MEM
+ help
+ Enable debug of all CXL command payloads.
+
+ Some CXL devices and controllers support encryption and other
+ security features. The payloads for the commands that enable
+ those features may contain sensitive clear-text security
+ material. Disable debug of those command payloads by default.
+ If you are a kernel developer actively working on CXL
+ security enabling say Y, otherwise say N.
endif
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 6d7d3870b5da..d63c8eaf23c7 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -346,6 +346,14 @@ static int __cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
/* #5 */
rc = cxl_mem_wait_for_doorbell(cxlm);
+
+ if (!cxl_is_security_command(mbox_cmd->opcode) ||
+ IS_ENABLED(CONFIG_CXL_MEM_INSECURE_DEBUG)) {
+ print_hex_dump_debug("Payload ", DUMP_PREFIX_OFFSET, 16, 1,
+mbox_cmd->payload_in, mbox_cmd->size_in,
+true);
+ }
+
if (rc == -ETIMEDOUT) {
cxl_mem_mbox_timeout(cxlm, mbox_cmd);
return rc;
--
2.30.1
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[PATCH v5 8/9] MAINTAINERS: Add maintainers of the CXL driver
Cc: Dan Williams Cc: Vishal Verma Cc: Ira Weiny Cc: Alison Schofield Signed-off-by: Ben Widawsky --- MAINTAINERS | 11 +++ 1 file changed, 11 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 6eff4f720c72..93c8694a8f04 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -,6 +,17 @@ M: Miguel Ojeda S: Maintained F: include/linux/compiler_attributes.h +COMPUTE EXPRESS LINK (CXL) +M: Alison Schofield +M: Vishal Verma +M: Ira Weiny +M: Ben Widawsky +M: Dan Williams +L: linux-...@vger.kernel.org +S: Maintained +F: drivers/cxl/ +F: include/uapi/linux/cxl_mem.h + CONEXANT ACCESSRUNNER USB DRIVER L: accessrunner-gene...@lists.sourceforge.net S: Orphan -- 2.30.1 ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[PATCH v5 7/9] cxl/mem: Add set of informational commands
Add initial set of formal commands beyond basic identify and command
enumeration.
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams
Reviewed-by: Jonathan Cameron (v2)
---
drivers/cxl/mem.c| 9 +
include/uapi/linux/cxl_mem.h | 5 +
2 files changed, 14 insertions(+)
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index e31b3045e231..6d7d3870b5da 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -45,12 +45,16 @@
enum opcode {
CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
+ CXL_MBOX_OP_GET_FW_INFO = 0x0200,
CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
CXL_MBOX_OP_GET_LOG = 0x0401,
CXL_MBOX_OP_IDENTIFY= 0x4000,
+ CXL_MBOX_OP_GET_PARTITION_INFO = 0x4100,
CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
+ CXL_MBOX_OP_GET_LSA = 0x4102,
CXL_MBOX_OP_SET_LSA = 0x4103,
+ CXL_MBOX_OP_GET_HEALTH_INFO = 0x4200,
CXL_MBOX_OP_SET_SHUTDOWN_STATE = 0x4204,
CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
@@ -171,6 +175,11 @@ static struct cxl_mem_command mem_commands[] = {
CXL_CMD(RAW, ~0, ~0, 0),
#endif
CXL_CMD(GET_SUPPORTED_LOGS, 0, ~0, CXL_CMD_FLAG_FORCE_ENABLE),
+ CXL_CMD(GET_FW_INFO, 0, 0x50, 0),
+ CXL_CMD(GET_PARTITION_INFO, 0, 0x20, 0),
+ CXL_CMD(GET_LSA, 0x8, ~0, 0),
+ CXL_CMD(GET_HEALTH_INFO, 0, 0x12, 0),
+ CXL_CMD(GET_LOG, 0x18, ~0, CXL_CMD_FLAG_FORCE_ENABLE),
};
/*
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index 59227f82a4c1..3155382dfc9b 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -24,6 +24,11 @@
___C(IDENTIFY, "Identify Command"), \
___C(RAW, "Raw device command"), \
___C(GET_SUPPORTED_LOGS, "Get Supported Logs"), \
+ ___C(GET_FW_INFO, "Get FW Info"), \
+ ___C(GET_PARTITION_INFO, "Get Partition Information"),\
+ ___C(GET_LSA, "Get Label Storage Area"), \
+ ___C(GET_HEALTH_INFO, "Get Health Info"), \
+ ___C(GET_LOG, "Get Log"), \
___C(MAX, "invalid / last command")
#define ___C(a, b) CXL_MEM_COMMAND_ID_##a
--
2.30.1
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[PATCH v5 6/9] cxl/mem: Enable commands via CEL
CXL devices identified by the memory-device class code must implement
the Device Command Interface (described in 8.2.9 of the CXL 2.0 spec).
While the driver already maintains a list of commands it supports, there
is still a need to be able to distinguish between commands that the
driver knows about from commands that are optionally supported by the
hardware.
The Command Effects Log (CEL) is specified in the CXL 2.0 specification.
The CEL is one of two types of logs, the other being vendor specific.
They are distinguished in hardware/spec via UUID. The CEL is useful for
2 things:
1. Determine which optional commands are supported by the CXL device.
2. Enumerate any vendor specific commands
The CEL is used by the driver to determine which commands are available
in the hardware and therefore which commands userspace is allowed to
execute. The set of enabled commands might be a subset of commands which
are advertised in UAPI via CXL_MEM_SEND_COMMAND IOCTL.
With the CEL enabling comes a internal flag to indicate a base set of
commands that are enabled regardless of CEL. Such commands are required
for basic interaction with the hardware and thus can be useful in debug
cases, for example if the CEL is corrupted.
The implementation leaves the statically defined table of commands and
supplements it with a bitmap to determine commands that are enabled.
This organization was chosen for the following reasons:
- Smaller memory footprint. Doesn't need a table per device.
- Reduce memory allocation complexity.
- Fixed command IDs to opcode mapping for all devices makes development
and debugging easier.
- Certain helpers are easily achievable, like cxl_for_each_cmd().
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams (v2)
Reviewed-by: Jonathan Cameron (v3)
---
drivers/cxl/cxl.h| 2 +
drivers/cxl/mem.c| 223 +--
include/uapi/linux/cxl_mem.h | 1 +
3 files changed, 219 insertions(+), 7 deletions(-)
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 8fd4a177fe25..6f14838c2d25 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -69,6 +69,7 @@ struct cxl_memdev;
*(CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
* @mbox_mutex: Mutex to synchronize mailbox access.
* @firmware_version: Firmware version for the memory device.
+ * @enabled_commands: Hardware commands found enabled in CEL.
* @pmem_range: Persistent memory capacity information.
* @ram_range: Volatile memory capacity information.
*/
@@ -84,6 +85,7 @@ struct cxl_mem {
size_t payload_size;
struct mutex mbox_mutex; /* Protects device mailbox and firmware */
char firmware_version[0x10];
+ unsigned long *enabled_cmds;
struct range pmem_range;
struct range ram_range;
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 5319412e245c..e31b3045e231 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -46,6 +46,8 @@ enum opcode {
CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
+ CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
+ CXL_MBOX_OP_GET_LOG = 0x0401,
CXL_MBOX_OP_IDENTIFY= 0x4000,
CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
CXL_MBOX_OP_SET_LSA = 0x4103,
@@ -108,10 +110,28 @@ static DEFINE_IDA(cxl_memdev_ida);
static struct dentry *cxl_debugfs;
static bool cxl_raw_allow_all;
+enum {
+ CEL_UUID,
+ VENDOR_DEBUG_UUID,
+};
+
+/* See CXL 2.0 Table 170. Get Log Input Payload */
+static const uuid_t log_uuid[] = {
+ [CEL_UUID] = UUID_INIT(0xda9c0b5, 0xbf41, 0x4b78, 0x8f, 0x79, 0x96,
+ 0xb1, 0x62, 0x3b, 0x3f, 0x17),
+ [VENDOR_DEBUG_UUID] = UUID_INIT(0xe1819d9, 0x11a9, 0x400c, 0x81, 0x1f,
+ 0xd6, 0x07, 0x19, 0x40, 0x3d, 0x86),
+};
+
/**
* struct cxl_mem_command - Driver representation of a memory device command
* @info: Command information as it exists for the UAPI
* @opcode: The actual bits used for the mailbox protocol
+ * @flags: Set of flags effecting driver behavior.
+ *
+ * * %CXL_CMD_FLAG_FORCE_ENABLE: In cases of error, commands with this flag
+ *will be enabled by the driver regardless of what hardware may have
+ *advertised.
*
* The cxl_mem_command is the driver's internal representation of commands that
* are supported by the driver. Some of these commands may not be supported by
@@ -123,9 +143,12 @@ static bool cxl_raw_allow_all;
struct cxl_mem_command {
struct cxl_command_info info;
enum opcode opcode;
+ u32 flags;
+#define CXL_CMD_FLAG_NONE 0
+#define CXL_CMD_FLAG_FORCE_ENABLE BIT(0)
};
-#define CXL_CMD(_id, sin, sout)
\
+#define CXL_CMD(_id, sin, sout, _flags)
\
[CXL_MEM_C
[PATCH v5 5/9] cxl/mem: Add a "RAW" send command
The CXL memory device send interface will have a number of supported
commands. The raw command is not such a command. Raw commands allow
userspace to send a specified opcode to the underlying hardware and
bypass all driver checks on the command. The primary use for this
command is to [begrudgingly] allow undocumented vendor specific hardware
commands.
While not the main motivation, it also allows prototyping new hardware
commands without a driver patch and rebuild.
While this all sounds very powerful it comes with a couple of caveats:
1. Bug reports using raw commands will not get the same level of
attention as bug reports using supported commands (via taint).
2. Supported commands will be rejected by the RAW command.
With this comes new debugfs knob to allow full access to your toes with
your weapon of choice.
Cc: Ariel Sibley
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams (v2)
Reviewed-by: Jonathan Cameron
---
drivers/cxl/Kconfig | 18 +
drivers/cxl/mem.c| 132 +++
include/uapi/linux/cxl_mem.h | 12 +++-
3 files changed, 161 insertions(+), 1 deletion(-)
diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
index 9e80b311e928..97dc4d751651 100644
--- a/drivers/cxl/Kconfig
+++ b/drivers/cxl/Kconfig
@@ -32,4 +32,22 @@ config CXL_MEM
Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification.
If unsure say 'm'.
+
+config CXL_MEM_RAW_COMMANDS
+ bool "RAW Command Interface for Memory Devices"
+ depends on CXL_MEM
+ help
+ Enable CXL RAW command interface.
+
+ The CXL driver ioctl interface may assign a kernel ioctl command
+ number for each specification defined opcode. At any given point in
+ time the number of opcodes that the specification defines and a device
+ may implement may exceed the kernel's set of associated ioctl function
+ numbers. The mismatch is either by omission, specification is too new,
+ or by design. When prototyping new hardware, or developing / debugging
+ the driver it is useful to be able to submit any possible command to
+ the hardware, even commands that may crash the kernel due to their
+ potential impact to memory currently in use by the kernel.
+
+ If developing CXL hardware or the driver say Y, otherwise say N.
endif
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index aa8f843fcca1..5319412e245c 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1,6 +1,8 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
#include
+#include
+#include
#include
#include
#include
@@ -42,7 +44,14 @@
enum opcode {
CXL_MBOX_OP_INVALID = 0x,
+ CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
+ CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
CXL_MBOX_OP_IDENTIFY= 0x4000,
+ CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
+ CXL_MBOX_OP_SET_LSA = 0x4103,
+ CXL_MBOX_OP_SET_SHUTDOWN_STATE = 0x4204,
+ CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
+ CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
CXL_MBOX_OP_MAX = 0x1
};
@@ -96,6 +105,8 @@ struct cxl_memdev {
static int cxl_mem_major;
static DEFINE_IDA(cxl_memdev_ida);
+static struct dentry *cxl_debugfs;
+static bool cxl_raw_allow_all;
/**
* struct cxl_mem_command - Driver representation of a memory device command
@@ -132,6 +143,49 @@ struct cxl_mem_command {
*/
static struct cxl_mem_command mem_commands[] = {
CXL_CMD(IDENTIFY, 0, 0x43),
+#ifdef CONFIG_CXL_MEM_RAW_COMMANDS
+ CXL_CMD(RAW, ~0, ~0),
+#endif
+};
+
+/*
+ * Commands that RAW doesn't permit. The rationale for each:
+ *
+ * CXL_MBOX_OP_ACTIVATE_FW: Firmware activation requires adjustment /
+ * coordination of transaction timeout values at the root bridge level.
+ *
+ * CXL_MBOX_OP_SET_PARTITION_INFO: The device memory map may change live
+ * and needs to be coordinated with HDM updates.
+ *
+ * CXL_MBOX_OP_SET_LSA: The label storage area may be cached by the
+ * driver and any writes from userspace invalidates those contents.
+ *
+ * CXL_MBOX_OP_SET_SHUTDOWN_STATE: Set shutdown state assumes no writes
+ * to the device after it is marked clean, userspace can not make that
+ * assertion.
+ *
+ * CXL_MBOX_OP_[GET_]SCAN_MEDIA: The kernel provides a native error list that
+ * is kept up to date with patrol notifications and error management.
+ */
+static u16 cxl_disabled_raw_commands[] = {
+ CXL_MBOX_OP_ACTIVATE_FW,
+ CXL_MBOX_OP_SET_PARTITION_INFO,
+ CXL_MBOX_OP_SET_LSA,
+ CXL_MBOX_OP_SET_SHUTDOWN_STATE,
+ CXL_MBOX_OP_SCAN_MEDIA,
+ CXL_MBOX_OP_GET_SCAN_MEDIA,
+};
+
+/*
+ * Command sets that RAW doesn't permit. All opcodes in this set are
+ * disabled because they pass plain text security payloads over
[PATCH v5 4/9] cxl/mem: Add basic IOCTL interface
Add a straightforward IOCTL that provides a mechanism for userspace to
query the supported memory device commands. CXL commands as they appear
to userspace are described as part of the UAPI kerneldoc. The command
list returned via this IOCTL will contain the full set of commands that
the driver supports, however, some of those commands may not be
available for use by userspace.
Memory device commands first appear in the CXL 2.0 specification. They
are submitted through a mailbox mechanism specified in the CXL 2.0
specification.
The send command allows userspace to issue mailbox commands directly to
the hardware. The list of available commands to send are the output of
the query command. The driver verifies basic properties of the command
and possibly inspect the input (or output) payload to determine whether
or not the command is allowed (or might taint the kernel).
Reported-by: kernel test robot # bug in earlier revision
Reported-by: Stephen Rothwell
Cc: Al Viro
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams (v2)
---
.clang-format | 1 +
.../driver-api/cxl/memory-devices.rst | 12 +
.../userspace-api/ioctl/ioctl-number.rst | 1 +
drivers/cxl/mem.c | 283 +-
include/uapi/linux/cxl_mem.h | 156 ++
5 files changed, 452 insertions(+), 1 deletion(-)
create mode 100644 include/uapi/linux/cxl_mem.h
diff --git a/.clang-format b/.clang-format
index 10dc5a9a61b3..3f11c8901b43 100644
--- a/.clang-format
+++ b/.clang-format
@@ -109,6 +109,7 @@ ForEachMacros:
- 'css_for_each_child'
- 'css_for_each_descendant_post'
- 'css_for_each_descendant_pre'
+ - 'cxl_for_each_cmd'
- 'device_for_each_child_node'
- 'dma_fence_chain_for_each'
- 'do_for_each_ftrace_op'
diff --git a/Documentation/driver-api/cxl/memory-devices.rst
b/Documentation/driver-api/cxl/memory-devices.rst
index 1fef2c0a167d..1bad466f9167 100644
--- a/Documentation/driver-api/cxl/memory-devices.rst
+++ b/Documentation/driver-api/cxl/memory-devices.rst
@@ -32,3 +32,15 @@ CXL Bus
---
.. kernel-doc:: drivers/cxl/bus.c
:doc: cxl bus
+
+External Interfaces
+===
+
+CXL IOCTL Interface
+---
+
+.. kernel-doc:: include/uapi/linux/cxl_mem.h
+ :doc: UAPI
+
+.. kernel-doc:: include/uapi/linux/cxl_mem.h
+ :internal:
diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst
b/Documentation/userspace-api/ioctl/ioctl-number.rst
index a4c75a28c839..6eb8e634664d 100644
--- a/Documentation/userspace-api/ioctl/ioctl-number.rst
+++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
@@ -352,6 +352,7 @@ Code Seq#Include File
Comments
<mailto:michael.kl...@puffin.lb.shuttle.de>
0xCC 00-0F drivers/misc/ibmvmc.h pseries
VMC driver
0xCD 01 linux/reiserfs_fs.h
+0xCE 01-02 uapi/linux/cxl_mem.hCompute
Express Link Memory Devices
0xCF 02 fs/cifs/ioctl.c
0xDB 00-0F drivers/char/mwave/mwavepub.h
0xDD 00-3F ZFCP
device driver see drivers/s390/scsi/
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 1c0195b07063..aa8f843fcca1 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include
#include
#include
#include
@@ -40,6 +41,7 @@
#define CXL_MAILBOX_TIMEOUT_MS (2 * HZ)
enum opcode {
+ CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_IDENTIFY= 0x4000,
CXL_MBOX_OP_MAX = 0x1
};
@@ -95,6 +97,49 @@ struct cxl_memdev {
static int cxl_mem_major;
static DEFINE_IDA(cxl_memdev_ida);
+/**
+ * struct cxl_mem_command - Driver representation of a memory device command
+ * @info: Command information as it exists for the UAPI
+ * @opcode: The actual bits used for the mailbox protocol
+ *
+ * The cxl_mem_command is the driver's internal representation of commands that
+ * are supported by the driver. Some of these commands may not be supported by
+ * the hardware. The driver will use @info to validate the fields passed in by
+ * the user then submit the @opcode to the hardware.
+ *
+ * See struct cxl_command_info.
+ */
+struct cxl_mem_command {
+ struct cxl_command_info info;
+ enum opcode opcode;
+};
+
+#define CXL_CMD(_id, sin, sout)
\
+ [CXL_MEM_COMMAND_ID_##_id] = { \
+ .info = { \
+ .id = CXL_MEM_
[PATCH v5 3/9] cxl/mem: Register CXL memX devices
From: Dan Williams
Create the /sys/bus/cxl hierarchy to enumerate:
* Memory Devices (per-endpoint control devices)
* Memory Address Space Devices (platform address ranges with
interleaving, performance, and persistence attributes)
* Memory Regions (active provisioned memory from an address space device
that is in use as System RAM or delegated to libnvdimm as Persistent
Memory regions).
For now, only the per-endpoint control devices are registered on the
'cxl' bus. However, going forward it will provide a mechanism to
coordinate cross-device interleave.
Signed-off-by: Dan Williams
Signed-off-by: Ben Widawsky
Reviewed-by: Jonathan Cameron (v2)
---
Documentation/ABI/testing/sysfs-bus-cxl | 26 ++
.../driver-api/cxl/memory-devices.rst | 5 +
drivers/cxl/Makefile | 3 +
drivers/cxl/bus.c | 29 ++
drivers/cxl/cxl.h | 3 +
drivers/cxl/mem.c | 285 +-
6 files changed, 349 insertions(+), 2 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl
create mode 100644 drivers/cxl/bus.c
diff --git a/Documentation/ABI/testing/sysfs-bus-cxl
b/Documentation/ABI/testing/sysfs-bus-cxl
new file mode 100644
index ..2fe7490ad6a8
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-bus-cxl
@@ -0,0 +1,26 @@
+What: /sys/bus/cxl/devices/memX/firmware_version
+Date: December, 2020
+KernelVersion: v5.12
+Contact: linux-...@vger.kernel.org
+Description:
+ (RO) "FW Revision" string as reported by the Identify
+ Memory Device Output Payload in the CXL-2.0
+ specification.
+
+What: /sys/bus/cxl/devices/memX/ram/size
+Date: December, 2020
+KernelVersion: v5.12
+Contact: linux-...@vger.kernel.org
+Description:
+ (RO) "Volatile Only Capacity" as bytes. Represents the
+ identically named field in the Identify Memory Device Output
+ Payload in the CXL-2.0 specification.
+
+What: /sys/bus/cxl/devices/memX/pmem/size
+Date: December, 2020
+KernelVersion: v5.12
+Contact: linux-...@vger.kernel.org
+Description:
+ (RO) "Persistent Only Capacity" as bytes. Represents the
+ identically named field in the Identify Memory Device Output
+ Payload in the CXL-2.0 specification.
diff --git a/Documentation/driver-api/cxl/memory-devices.rst
b/Documentation/driver-api/cxl/memory-devices.rst
index 43177e700d62..1fef2c0a167d 100644
--- a/Documentation/driver-api/cxl/memory-devices.rst
+++ b/Documentation/driver-api/cxl/memory-devices.rst
@@ -27,3 +27,8 @@ CXL Memory Device
.. kernel-doc:: drivers/cxl/mem.c
:internal:
+
+CXL Bus
+---
+.. kernel-doc:: drivers/cxl/bus.c
+ :doc: cxl bus
diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile
index 4a30f7c3fc4a..a314a1891f4d 100644
--- a/drivers/cxl/Makefile
+++ b/drivers/cxl/Makefile
@@ -1,4 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_CXL_BUS) += cxl_bus.o
obj-$(CONFIG_CXL_MEM) += cxl_mem.o
+ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
+cxl_bus-y := bus.o
cxl_mem-y := mem.o
diff --git a/drivers/cxl/bus.c b/drivers/cxl/bus.c
new file mode 100644
index ..58f74796d525
--- /dev/null
+++ b/drivers/cxl/bus.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include
+#include
+
+/**
+ * DOC: cxl bus
+ *
+ * The CXL bus provides namespace for control devices and a rendezvous
+ * point for cross-device interleave coordination.
+ */
+struct bus_type cxl_bus_type = {
+ .name = "cxl",
+};
+EXPORT_SYMBOL_GPL(cxl_bus_type);
+
+static __init int cxl_bus_init(void)
+{
+ return bus_register(&cxl_bus_type);
+}
+
+static void cxl_bus_exit(void)
+{
+ bus_unregister(&cxl_bus_type);
+}
+
+module_init(cxl_bus_init);
+module_exit(cxl_bus_exit);
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index baac26d9e63b..8fd4a177fe25 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -57,6 +57,7 @@
(FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \
CXLMDEV_RESET_NEEDED_NOT)
+struct cxl_memdev;
/**
* struct cxl_mem - A CXL memory device
* @pdev: The PCI device associated with this CXL device.
@@ -74,6 +75,7 @@
struct cxl_mem {
struct pci_dev *pdev;
void __iomem *regs;
+ struct cxl_memdev *cxlmd;
void __iomem *status_regs;
void __iomem *mbox_regs;
@@ -87,4 +89,5 @@ struct cxl_mem {
struct range ram_range;
};
+extern struct bus_type cxl_bus_type;
#endif /* __CXL_H__ */
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 04e15759bc9b..1c0195b07063 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1,6 +1,9 @@
[PATCH v5 2/9] cxl/mem: Find device capabilities
Provide enough functionality to utilize the mailbox of a memory device. The mailbox is used to interact with the firmware running on the memory device. The flow is proven with one implemented command, "identify". Because the class code has already told the driver this is a memory device and the identify command is mandatory. CXL devices contain an array of capabilities that describe the interactions software can have with the device or firmware running on the device. A CXL compliant device must implement the device status and the mailbox capability. Additionally, a CXL compliant memory device must implement the memory device capability. Each of the capabilities can [will] provide an offset within the MMIO region for interacting with the CXL device. The capabilities tell the driver how to find and map the register space for CXL Memory Devices. The registers are required to utilize the CXL spec defined mailbox interface. The spec outlines two mailboxes, primary and secondary. The secondary mailbox is earmarked for system firmware, and not handled in this driver. Primary mailboxes are capable of generating an interrupt when submitting a background command. That implementation is saved for a later time. Reported-by: Colin Ian King (coverity) Reported-by: Dan Carpenter (smatch) Link: https://www.computeexpresslink.org/download-the-specification Signed-off-by: Ben Widawsky Reviewed-by: Dan Williams (v2) --- .../driver-api/cxl/memory-devices.rst | 14 + drivers/cxl/cxl.h | 90 +++ drivers/cxl/mem.c | 577 +- drivers/cxl/pci.h | 14 + 4 files changed, 693 insertions(+), 2 deletions(-) create mode 100644 drivers/cxl/cxl.h diff --git a/Documentation/driver-api/cxl/memory-devices.rst b/Documentation/driver-api/cxl/memory-devices.rst index 71617e9002f1..43177e700d62 100644 --- a/Documentation/driver-api/cxl/memory-devices.rst +++ b/Documentation/driver-api/cxl/memory-devices.rst @@ -13,3 +13,17 @@ Address space is handled via HDM (Host Managed Device Memory) decoders that optionally define a device's contribution to an interleaved address range across multiple devices underneath a host-bridge or interleaved across host-bridges. + +Driver Infrastructure += + +This section covers the driver infrastructure for a CXL memory device. + +CXL Memory Device +- + +.. kernel-doc:: drivers/cxl/mem.c + :doc: cxl mem + +.. kernel-doc:: drivers/cxl/mem.c + :internal: diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h new file mode 100644 index ..baac26d9e63b --- /dev/null +++ b/drivers/cxl/cxl.h @@ -0,0 +1,90 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2020 Intel Corporation. */ + +#ifndef __CXL_H__ +#define __CXL_H__ + +#include +#include +#include + +/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */ +#define CXLDEV_CAP_ARRAY_OFFSET 0x0 +#define CXLDEV_CAP_ARRAY_CAP_ID 0 +#define CXLDEV_CAP_ARRAY_ID_MASK GENMASK_ULL(15, 0) +#define CXLDEV_CAP_ARRAY_COUNT_MASK GENMASK_ULL(47, 32) +/* CXL 2.0 8.2.8.2 CXL Device Capability Header Register */ +#define CXLDEV_CAP_HDR_CAP_ID_MASK GENMASK(15, 0) +/* CXL 2.0 8.2.8.2.1 CXL Device Capabilities */ +#define CXLDEV_CAP_CAP_ID_DEVICE_STATUS 0x1 +#define CXLDEV_CAP_CAP_ID_PRIMARY_MAILBOX 0x2 +#define CXLDEV_CAP_CAP_ID_SECONDARY_MAILBOX 0x3 +#define CXLDEV_CAP_CAP_ID_MEMDEV 0x4000 + +/* CXL 2.0 8.2.8.4 Mailbox Registers */ +#define CXLDEV_MBOX_CAPS_OFFSET 0x00 +#define CXLDEV_MBOX_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0) +#define CXLDEV_MBOX_CTRL_OFFSET 0x04 +#define CXLDEV_MBOX_CTRL_DOORBELL BIT(0) +#define CXLDEV_MBOX_CMD_OFFSET 0x08 +#define CXLDEV_MBOX_CMD_COMMAND_OPCODE_MASK GENMASK_ULL(15, 0) +#define CXLDEV_MBOX_CMD_PAYLOAD_LENGTH_MASK GENMASK_ULL(36, 16) +#define CXLDEV_MBOX_STATUS_OFFSET 0x10 +#define CXLDEV_MBOX_STATUS_RET_CODE_MASK GENMASK_ULL(47, 32) +#define CXLDEV_MBOX_BG_CMD_STATUS_OFFSET 0x18 +#define CXLDEV_MBOX_PAYLOAD_OFFSET 0x20 + +/* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */ +#define CXLMDEV_STATUS_OFFSET 0x0 +#define CXLMDEV_DEV_FATAL BIT(0) +#define CXLMDEV_FW_HALT BIT(1) +#define CXLMDEV_STATUS_MEDIA_STATUS_MASK GENMASK(3, 2) +#define CXLMDEV_MS_NOT_READY 0 +#define CXLMDEV_MS_READY 1 +#define CXLMDEV_MS_ERROR 2 +#define CXLMDEV_MS_DISABLED 3 +#define CXLMDEV_READY(status) \ + (FIELD_GET(CXLMDEV_STATUS_MEDIA_STATUS_MASK, status) ==\ +CXLMDEV_MS_READY) +#define CXLMDEV_MBOX_IF_READY BIT(4) +#define CXLMDEV_RESET_NEEDED_MASK GENMASK(7, 5) +#define CXLMDEV_RESET_NEEDED_NOT 0 +#define CXLMDEV_RESET_NEEDED_COLD 1 +#define CXLMDEV_RESET_NEEDED_WARM 2 +#define CXLMDEV_RESET_NEEDED_HOT 3 +#define CXLMDEV_RESET_NEEDED_CXL 4 +#define CXLMDEV_RESET_NEEDED(status) \ +
[PATCH v5 1/9] cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints
From: Dan Williams The CXL.mem protocol allows a device to act as a provider of "System RAM" and/or "Persistent Memory" that is fully coherent as if the memory was attached to the typical CPU memory controller. With the CXL-2.0 specification a PCI endpoint can implement a "Type-3" device interface and give the operating system control over "Host Managed Device Memory". See section 2.3 Type 3 CXL Device. The memory range exported by the device may optionally be described by the platform firmware memory map, or by infrastructure like LIBNVDIMM to provision persistent memory capacity from one, or more, CXL.mem devices. A pre-requisite for Linux-managed memory-capacity provisioning is this cxl_mem driver that can speak the mailbox protocol defined in section 8.2.8.4 Mailbox Registers. For now just land the initial driver boiler-plate and Documentation/ infrastructure. Link: https://www.computeexpresslink.org/download-the-specification Cc: Jonathan Corbet Signed-off-by: Dan Williams Signed-off-by: Ben Widawsky Acked-by: David Rientjes (v1) Reviewed-by: Jonathan Cameron --- Documentation/driver-api/cxl/index.rst| 12 .../driver-api/cxl/memory-devices.rst | 15 + Documentation/driver-api/index.rst| 1 + drivers/Kconfig | 1 + drivers/Makefile | 1 + drivers/cxl/Kconfig | 35 +++ drivers/cxl/Makefile | 4 ++ drivers/cxl/mem.c | 62 +++ drivers/cxl/pci.h | 17 + include/linux/pci_ids.h | 1 + 10 files changed, 149 insertions(+) create mode 100644 Documentation/driver-api/cxl/index.rst create mode 100644 Documentation/driver-api/cxl/memory-devices.rst create mode 100644 drivers/cxl/Kconfig create mode 100644 drivers/cxl/Makefile create mode 100644 drivers/cxl/mem.c create mode 100644 drivers/cxl/pci.h diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst new file mode 100644 index ..036e49553542 --- /dev/null +++ b/Documentation/driver-api/cxl/index.rst @@ -0,0 +1,12 @@ +.. SPDX-License-Identifier: GPL-2.0 + + +Compute Express Link + + +.. toctree:: + :maxdepth: 1 + + memory-devices + +.. only:: subproject and html diff --git a/Documentation/driver-api/cxl/memory-devices.rst b/Documentation/driver-api/cxl/memory-devices.rst new file mode 100644 index ..71617e9002f1 --- /dev/null +++ b/Documentation/driver-api/cxl/memory-devices.rst @@ -0,0 +1,15 @@ +.. SPDX-License-Identifier: GPL-2.0 +.. include:: + +=== +Compute Express Link Memory Devices +=== + +A Compute Express Link Memory Device is a CXL component that implements the +CXL.mem protocol. It contains some amount of volatile memory, persistent memory, +or both. It is enumerated as a PCI device for configuration and passing +messages over an MMIO mailbox. Its contribution to the System Physical +Address space is handled via HDM (Host Managed Device Memory) decoders +that optionally define a device's contribution to an interleaved address +range across multiple devices underneath a host-bridge or interleaved +across host-bridges. diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst index 2456d0a97ed8..d246a18fd78f 100644 --- a/Documentation/driver-api/index.rst +++ b/Documentation/driver-api/index.rst @@ -35,6 +35,7 @@ available subsections can be seen below. usb/index firewire pci/index + cxl/index spi i2c ipmb diff --git a/drivers/Kconfig b/drivers/Kconfig index dcecc9f6e33f..62c753a73651 100644 --- a/drivers/Kconfig +++ b/drivers/Kconfig @@ -6,6 +6,7 @@ menu "Device Drivers" source "drivers/amba/Kconfig" source "drivers/eisa/Kconfig" source "drivers/pci/Kconfig" +source "drivers/cxl/Kconfig" source "drivers/pcmcia/Kconfig" source "drivers/rapidio/Kconfig" diff --git a/drivers/Makefile b/drivers/Makefile index fd11b9ac4cc3..678ea810410f 100644 --- a/drivers/Makefile +++ b/drivers/Makefile @@ -73,6 +73,7 @@ obj-$(CONFIG_NVM) += lightnvm/ obj-y += base/ block/ misc/ mfd/ nfc/ obj-$(CONFIG_LIBNVDIMM)+= nvdimm/ obj-$(CONFIG_DAX) += dax/ +obj-$(CONFIG_CXL_BUS) += cxl/ obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/ obj-$(CONFIG_NUBUS)+= nubus/ obj-y += macintosh/ diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig new file mode 100644 index ..9e80b311e928 --- /dev/null +++ b/drivers/cxl/Kconfig @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: GPL-2.0-only +menuconfig CXL_BUS + tristate "CXL (Compute Express Li
[PATCH v5 0/9] CXL 2.0 Support
-+ +-+ // Memory backend for "window" -object memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M // Memory backend for LSA -object memory-backend-file,id=cxl-mem1-lsa,share,mem-path=cxl-mem1-lsa,size=1K // Host Bridge -device pxb-cxl id=cxl.0,bus=pcie.0,bus_nr=52,uid=0 len-window-base=1,window-base[0]=0x4c000 memdev[0]=cxl-mem1 // Single root port -device cxl rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0,memdev=cxl-mem1 // Single type3 device -device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M -device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M,lsa=cxl-mem1-lsa --- [1]: https://lore.kernel.org/linux-cxl/20210216014538.268106-1-ben.widaw...@intel.com/ [2]: https://www.computeexpresslink.org/](https://www.computeexpresslink.org/ [3]: https://lore.kernel.org/qemu-devel/20210202005948.241655-1-ben.widaw...@intel.com/ [4]: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v4 [5]: https://github.com/pmem/ndctl/tree/cxl-2.0v2 Cc: linux-a...@vger.kernel.org Cc: linux-ker...@vger.kernel.org Cc: linux-nvdimm@lists.01.org Cc: linux-...@vger.kernel.org Cc: Bjorn Helgaas Cc: Chris Browy Cc: Christoph Hellwig Cc: Dan Williams Cc: David Hildenbrand Cc: David Rientjes Cc: Ira Weiny Cc: Jon Masters Cc: Jonathan Cameron Cc: Rafael Wysocki Cc: Randy Dunlap Cc: Vishal Verma Cc: "John Groves (jgroves)" Cc: "Kelley, Sean V" --- Ben Widawsky (7): cxl/mem: Find device capabilities cxl/mem: Add basic IOCTL interface cxl/mem: Add a "RAW" send command cxl/mem: Enable commands via CEL cxl/mem: Add set of informational commands MAINTAINERS: Add maintainers of the CXL driver cxl/mem: Add payload dumping for debug Dan Williams (2): cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints cxl/mem: Register CXL memX devices .clang-format |1 + Documentation/ABI/testing/sysfs-bus-cxl | 26 + Documentation/driver-api/cxl/index.rst| 12 + .../driver-api/cxl/memory-devices.rst | 46 + Documentation/driver-api/index.rst|1 + .../userspace-api/ioctl/ioctl-number.rst |1 + MAINTAINERS | 11 + drivers/Kconfig |1 + drivers/Makefile |1 + drivers/cxl/Kconfig | 66 + drivers/cxl/Makefile |7 + drivers/cxl/bus.c | 29 + drivers/cxl/cxl.h | 95 + drivers/cxl/mem.c | 1553 + drivers/cxl/pci.h | 31 + include/linux/pci_ids.h |1 + include/uapi/linux/cxl_mem.h | 172 ++ 17 files changed, 2054 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl create mode 100644 Documentation/driver-api/cxl/index.rst create mode 100644 Documentation/driver-api/cxl/memory-devices.rst create mode 100644 drivers/cxl/Kconfig create mode 100644 drivers/cxl/Makefile create mode 100644 drivers/cxl/bus.c create mode 100644 drivers/cxl/cxl.h create mode 100644 drivers/cxl/mem.c create mode 100644 drivers/cxl/pci.h create mode 100644 include/uapi/linux/cxl_mem.h -- 2.30.1 ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH v4 4/9] cxl/mem: Add basic IOCTL interface
On 21-02-16 18:12:05, Al Viro wrote:
> On Mon, Feb 15, 2021 at 05:45:33PM -0800, Ben Widawsky wrote:
> > + if (cmd->info.size_in) {
> > + mbox_cmd.payload_in = kvzalloc(cmd->info.size_in, GFP_KERNEL);
> > + if (!mbox_cmd.payload_in) {
> > + rc = -ENOMEM;
> > + goto out;
> > + }
> > +
> > + if (copy_from_user(mbox_cmd.payload_in,
> > + u64_to_user_ptr(in_payload),
> > + cmd->info.size_in)) {
> > + rc = -EFAULT;
> > + goto out;
> > + }
>
> Umm... Do you need to open-code vmemdup_user()? The only difference is
> GFP_KERNEL allocation instead of GFP_USER one, and the latter is arguably
> saner here... Zeroing is definitely pointless - you either overwrite
> the entire buffer with copy_from_user(), or you fail and free the damn
> thing.
mea culpa. In fact it was previously memdup_user and Dan suggested I switch to
vmemdup_user.
https://lore.kernel.org/linux-cxl/capcyv4j+ixvgeo5q2ohv4kdkbzbnohzj3kdovreqjjpbsre...@mail.gmail.com/
Will fix for the next version.
Thanks.
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH v4 4/9] cxl/mem: Add basic IOCTL interface
On 21-02-16 18:28:49, Jonathan Cameron wrote: > On Tue, 16 Feb 2021 09:53:14 -0800 > Ben Widawsky wrote: > > > On 21-02-16 15:22:23, Jonathan Cameron wrote: > > > On Mon, 15 Feb 2021 17:45:33 -0800 > > > Ben Widawsky wrote: > > > > > > > Add a straightforward IOCTL that provides a mechanism for userspace to > > > > query the supported memory device commands. CXL commands as they appear > > > > to userspace are described as part of the UAPI kerneldoc. The command > > > > list returned via this IOCTL will contain the full set of commands that > > > > the driver supports, however, some of those commands may not be > > > > available for use by userspace. > > > > > > > > Memory device commands first appear in the CXL 2.0 specification. They > > > > are submitted through a mailbox mechanism specified in the CXL 2.0 > > > > specification. > > > > > > > > The send command allows userspace to issue mailbox commands directly to > > > > the hardware. The list of available commands to send are the output of > > > > the query command. The driver verifies basic properties of the command > > > > and possibly inspect the input (or output) payload to determine whether > > > > or not the command is allowed (or might taint the kernel). > > > > > > > > Reported-by: kernel test robot # bug in earlier > > > > revision > > > > Signed-off-by: Ben Widawsky > > > > Reviewed-by: Dan Williams (v2) > > > > > > I may be missreading this but I think the logic to ensure commands > > > using a variable sized buffer have enough space is broken. > > > > > > Jonathan > > > > > > > --- > > > > .clang-format | 1 + > > > > .../userspace-api/ioctl/ioctl-number.rst | 1 + > > > > drivers/cxl/mem.c | 288 +- > > > > include/uapi/linux/cxl_mem.h | 154 ++ > > > > 4 files changed, 443 insertions(+), 1 deletion(-) > > > > create mode 100644 include/uapi/linux/cxl_mem.h > > > > > > > > diff --git a/.clang-format b/.clang-format > > > > index 10dc5a9a61b3..3f11c8901b43 100644 > > > > --- a/.clang-format > > > > +++ b/.clang-format > > > > @@ -109,6 +109,7 @@ ForEachMacros: > > > >- 'css_for_each_child' > > > >- 'css_for_each_descendant_post' > > > >- 'css_for_each_descendant_pre' > > > > + - 'cxl_for_each_cmd' > > > >- 'device_for_each_child_node' > > > >- 'dma_fence_chain_for_each' > > > >- 'do_for_each_ftrace_op' > > > > diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst > > > > b/Documentation/userspace-api/ioctl/ioctl-number.rst > > > > index a4c75a28c839..6eb8e634664d 100644 > > > > --- a/Documentation/userspace-api/ioctl/ioctl-number.rst > > > > +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst > > > > @@ -352,6 +352,7 @@ Code Seq#Include File > > > > Comments > > > > > > > > <mailto:michael.kl...@puffin.lb.shuttle.de> > > > > 0xCC 00-0F drivers/misc/ibmvmc.h > > > > pseries VMC driver > > > > 0xCD 01 linux/reiserfs_fs.h > > > > +0xCE 01-02 uapi/linux/cxl_mem.h > > > > Compute Express Link Memory Devices > > > > 0xCF 02 fs/cifs/ioctl.c > > > > 0xDB 00-0F drivers/char/mwave/mwavepub.h > > > > 0xDD 00-3F > > > > ZFCP device driver see drivers/s390/scsi/ > > > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c > > > > index 410adb1bdffc..a4298cb1182d 100644 > > > > --- a/drivers/cxl/mem.c > > > > +++ b/drivers/cxl/mem.c > > > > @@ -1,5 +1,6 @@ > > > > // SPDX-License-Identifier: GPL-2.0-only > > > > /* Copyright(c) 2020 Intel Corporation. All rights reserved. */ > > > > +#include > > > > #include > > > > #include > > > > #include > > > > @@ -40,6 +41,7 @@ > > > > #define CXL_MAIL
Re: [PATCH v4 5/9] cxl/mem: Add a "RAW" send command
On 21-02-16 15:30:26, Jonathan Cameron wrote:
> On Mon, 15 Feb 2021 17:45:34 -0800
> Ben Widawsky wrote:
>
> > The CXL memory device send interface will have a number of supported
> > commands. The raw command is not such a command. Raw commands allow
> > userspace to send a specified opcode to the underlying hardware and
> > bypass all driver checks on the command. The primary use for this
> > command is to [begrudgingly] allow undocumented vendor specific hardware
> > commands.
> >
> > While not the main motivation, it also allows prototyping new hardware
> > commands without a driver patch and rebuild.
> >
> > While this all sounds very powerful it comes with a couple of caveats:
> > 1. Bug reports using raw commands will not get the same level of
> >attention as bug reports using supported commands (via taint).
> > 2. Supported commands will be rejected by the RAW command.
> >
> > With this comes new debugfs knob to allow full access to your toes with
> > your weapon of choice.
> >
> > Cc: Ariel Sibley
> > Signed-off-by: Ben Widawsky
> > Reviewed-by: Dan Williams (v2)
>
> Whilst I'm definitely dubious about introducing this interface
> so early in development, I haven't found any problems with 'how' it
> has been done.
>
FWIW, it's already helpful for regression testing. ndctl/cxl(1) will make use of
it for validating driver internals and our QEMU emulation. I don't mean to imply
that's the only usage.
> I guess it's now just up to us to hassle our hardware colleagues into
> only using this facility when absolutely necessary...
Yes. I think having distros not enable the Kconfig option is going to be the
best path in the early days so they don't get used to having it available.
>
> Reviewed-by: Jonathan Cameron
>
> > ---
> > drivers/cxl/Kconfig | 18 +
> > drivers/cxl/mem.c| 132 +++
> > include/uapi/linux/cxl_mem.h | 12 +++-
> > 3 files changed, 161 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
> > index 9e80b311e928..97dc4d751651 100644
> > --- a/drivers/cxl/Kconfig
> > +++ b/drivers/cxl/Kconfig
> > @@ -32,4 +32,22 @@ config CXL_MEM
> > Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification.
> >
> > If unsure say 'm'.
> > +
> > +config CXL_MEM_RAW_COMMANDS
> > + bool "RAW Command Interface for Memory Devices"
> > + depends on CXL_MEM
> > + help
> > + Enable CXL RAW command interface.
> > +
> > + The CXL driver ioctl interface may assign a kernel ioctl command
> > + number for each specification defined opcode. At any given point in
> > + time the number of opcodes that the specification defines and a device
> > + may implement may exceed the kernel's set of associated ioctl function
> > + numbers. The mismatch is either by omission, specification is too new,
> > + or by design. When prototyping new hardware, or developing / debugging
> > + the driver it is useful to be able to submit any possible command to
> > + the hardware, even commands that may crash the kernel due to their
> > + potential impact to memory currently in use by the kernel.
> > +
> > + If developing CXL hardware or the driver say Y, otherwise say N.
> > endif
> > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > index a4298cb1182d..6b4feb0ce47d 100644
> > --- a/drivers/cxl/mem.c
> > +++ b/drivers/cxl/mem.c
> > @@ -1,6 +1,8 @@
> > // SPDX-License-Identifier: GPL-2.0-only
> > /* Copyright(c) 2020 Intel Corporation. All rights reserved. */
> > #include
> > +#include
> > +#include
> > #include
> > #include
> > #include
> > @@ -42,7 +44,14 @@
> >
> > enum opcode {
> > CXL_MBOX_OP_INVALID = 0x,
> > + CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
> > + CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
> > CXL_MBOX_OP_IDENTIFY= 0x4000,
> > + CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
> > + CXL_MBOX_OP_SET_LSA = 0x4103,
> > + CXL_MBOX_OP_SET_SHUTDOWN_STATE = 0x4204,
> > + CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
> > + CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
> > CXL_MBOX_OP_MAX = 0x1
> > };
> >
> > @@ -92,6 +101,8 @@ struct cxl_memdev {
> >
> > static int cxl_mem_major;
> > static DEFINE_IDA(cxl_memdev_ida);
> >
Re: [PATCH v4 2/9] cxl/mem: Find device capabilities
On 21-02-16 17:20:01, Jonathan Cameron wrote:
> On Tue, 16 Feb 2021 08:43:03 -0800
> Ben Widawsky wrote:
>
> > On 21-02-16 14:51:48, Jonathan Cameron wrote:
> > > On Mon, 15 Feb 2021 17:45:31 -0800
> > > Ben Widawsky wrote:
> > >
> > > > Provide enough functionality to utilize the mailbox of a memory device.
> > > > The mailbox is used to interact with the firmware running on the memory
> > > > device. The flow is proven with one implemented command, "identify".
> > > > Because the class code has already told the driver this is a memory
> > > > device and the identify command is mandatory.
> > > >
> > > > CXL devices contain an array of capabilities that describe the
> > > > interactions software can have with the device or firmware running on
> > > > the device. A CXL compliant device must implement the device status and
> > > > the mailbox capability. Additionally, a CXL compliant memory device must
> > > > implement the memory device capability. Each of the capabilities can
> > > > [will] provide an offset within the MMIO region for interacting with the
> > > > CXL device.
> > > >
> > > > The capabilities tell the driver how to find and map the register space
> > > > for CXL Memory Devices. The registers are required to utilize the CXL
> > > > spec defined mailbox interface. The spec outlines two mailboxes, primary
> > > > and secondary. The secondary mailbox is earmarked for system firmware,
> > > > and not handled in this driver.
> > > >
> > > > Primary mailboxes are capable of generating an interrupt when submitting
> > > > a background command. That implementation is saved for a later time.
> > > >
> > > > Link: https://www.computeexpresslink.org/download-the-specification
> > > > Signed-off-by: Ben Widawsky
> > > > Reviewed-by: Dan Williams (v2)
> > >
> > > Looks like an off by one error in the register locator iteration.
> > >
> > > The potential buffer overrun from memcpy_fromio is still there as well
> > > as far as I can see.
> > >
> > > If the software provides storage for a payload of size n and the hardware
> > > reports a size of n + d, code will happily write beyond the end of the
> > > storage provided.
> > >
> > > Obviously, this shouldn't happen, but I'm not that trusting of both
> > > hardware and software never having bugs.
> > >
> > > Jonathan
> > >
> > > > ---
> > > > drivers/cxl/cxl.h | 88
> > > > drivers/cxl/mem.c | 543 +-
> > > > drivers/cxl/pci.h | 14 ++
> > > > 3 files changed, 643 insertions(+), 2 deletions(-)
> > > > create mode 100644 drivers/cxl/cxl.h
> > > >
> > > ...
> > >
> > > > +
> > > > +#endif /* __CXL_H__ */
> > > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > > > index ce33c5ee77c9..b86cda2d299a 100644
> > > > --- a/drivers/cxl/mem.c
> > > > +++ b/drivers/cxl/mem.c
> > > > @@ -3,7 +3,458 @@
> > > > #include
> > > > #include
> > > > #include
> > > > +#include
> > > > #include "pci.h"
> > > > +#include "cxl.h"
> > > > +
> > > > +#define cxl_doorbell_busy(cxlm)
> > > > \
> > > > + (readl((cxlm)->mbox_regs + CXLDEV_MBOX_CTRL_OFFSET) &
> > > >\
> > > > +CXLDEV_MBOX_CTRL_DOORBELL)
> > > > +
> > > > +/* CXL 2.0 - 8.2.8.4 */
> > > > +#define CXL_MAILBOX_TIMEOUT_MS (2 * HZ)
> > > > +
> > > > +enum opcode {
> > > > + CXL_MBOX_OP_IDENTIFY= 0x4000,
> > > > + CXL_MBOX_OP_MAX = 0x1
> > > > +};
> > > > +
> > > > +/**
> > > > + * struct mbox_cmd - A command to be submitted to hardware.
> > > > + * @opcode: (input) The command set and command submitted to hardware.
> > > > + * @payload_in: (input) Pointer to the input payload.
> > > > + * @payload_out: (output) Pointer to the output payload. Must be
> > > > allocated by
> > > > + * the caller.
> > > > + * @size_in: (input) Number of byte
Re: [PATCH v4 4/9] cxl/mem: Add basic IOCTL interface
On 21-02-16 15:22:23, Jonathan Cameron wrote:
> On Mon, 15 Feb 2021 17:45:33 -0800
> Ben Widawsky wrote:
>
> > Add a straightforward IOCTL that provides a mechanism for userspace to
> > query the supported memory device commands. CXL commands as they appear
> > to userspace are described as part of the UAPI kerneldoc. The command
> > list returned via this IOCTL will contain the full set of commands that
> > the driver supports, however, some of those commands may not be
> > available for use by userspace.
> >
> > Memory device commands first appear in the CXL 2.0 specification. They
> > are submitted through a mailbox mechanism specified in the CXL 2.0
> > specification.
> >
> > The send command allows userspace to issue mailbox commands directly to
> > the hardware. The list of available commands to send are the output of
> > the query command. The driver verifies basic properties of the command
> > and possibly inspect the input (or output) payload to determine whether
> > or not the command is allowed (or might taint the kernel).
> >
> > Reported-by: kernel test robot # bug in earlier revision
> > Signed-off-by: Ben Widawsky
> > Reviewed-by: Dan Williams (v2)
>
> I may be missreading this but I think the logic to ensure commands
> using a variable sized buffer have enough space is broken.
>
> Jonathan
>
> > ---
> > .clang-format | 1 +
> > .../userspace-api/ioctl/ioctl-number.rst | 1 +
> > drivers/cxl/mem.c | 288 +-
> > include/uapi/linux/cxl_mem.h | 154 ++
> > 4 files changed, 443 insertions(+), 1 deletion(-)
> > create mode 100644 include/uapi/linux/cxl_mem.h
> >
> > diff --git a/.clang-format b/.clang-format
> > index 10dc5a9a61b3..3f11c8901b43 100644
> > --- a/.clang-format
> > +++ b/.clang-format
> > @@ -109,6 +109,7 @@ ForEachMacros:
> >- 'css_for_each_child'
> >- 'css_for_each_descendant_post'
> >- 'css_for_each_descendant_pre'
> > + - 'cxl_for_each_cmd'
> >- 'device_for_each_child_node'
> >- 'dma_fence_chain_for_each'
> >- 'do_for_each_ftrace_op'
> > diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst
> > b/Documentation/userspace-api/ioctl/ioctl-number.rst
> > index a4c75a28c839..6eb8e634664d 100644
> > --- a/Documentation/userspace-api/ioctl/ioctl-number.rst
> > +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
> > @@ -352,6 +352,7 @@ Code Seq#Include File
> > Comments
> >
> > <mailto:michael.kl...@puffin.lb.shuttle.de>
> > 0xCC 00-0F drivers/misc/ibmvmc.h
> > pseries VMC driver
> > 0xCD 01 linux/reiserfs_fs.h
> > +0xCE 01-02 uapi/linux/cxl_mem.h
> > Compute Express Link Memory Devices
> > 0xCF 02 fs/cifs/ioctl.c
> > 0xDB 00-0F drivers/char/mwave/mwavepub.h
> > 0xDD 00-3F ZFCP
> > device driver see drivers/s390/scsi/
> > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > index 410adb1bdffc..a4298cb1182d 100644
> > --- a/drivers/cxl/mem.c
> > +++ b/drivers/cxl/mem.c
> > @@ -1,5 +1,6 @@
> > // SPDX-License-Identifier: GPL-2.0-only
> > /* Copyright(c) 2020 Intel Corporation. All rights reserved. */
> > +#include
> > #include
> > #include
> > #include
> > @@ -40,6 +41,7 @@
> > #define CXL_MAILBOX_TIMEOUT_MS (2 * HZ)
> >
> > enum opcode {
> > + CXL_MBOX_OP_INVALID = 0x,
> > CXL_MBOX_OP_IDENTIFY= 0x4000,
> > CXL_MBOX_OP_MAX = 0x1
> > };
> > @@ -91,6 +93,49 @@ struct cxl_memdev {
> > static int cxl_mem_major;
> > static DEFINE_IDA(cxl_memdev_ida);
> >
> > +/**
> > + * struct cxl_mem_command - Driver representation of a memory device
> > command
> > + * @info: Command information as it exists for the UAPI
> > + * @opcode: The actual bits used for the mailbox protocol
> > + *
> > + * The cxl_mem_command is the driver's internal representation of commands
> > that
> > + * are supported by the driver. Some of these commands may not be
> > supported by
> > + * the hardware. T
Re: [PATCH v4 2/9] cxl/mem: Find device capabilities
On 21-02-16 14:51:48, Jonathan Cameron wrote:
> On Mon, 15 Feb 2021 17:45:31 -0800
> Ben Widawsky wrote:
>
> > Provide enough functionality to utilize the mailbox of a memory device.
> > The mailbox is used to interact with the firmware running on the memory
> > device. The flow is proven with one implemented command, "identify".
> > Because the class code has already told the driver this is a memory
> > device and the identify command is mandatory.
> >
> > CXL devices contain an array of capabilities that describe the
> > interactions software can have with the device or firmware running on
> > the device. A CXL compliant device must implement the device status and
> > the mailbox capability. Additionally, a CXL compliant memory device must
> > implement the memory device capability. Each of the capabilities can
> > [will] provide an offset within the MMIO region for interacting with the
> > CXL device.
> >
> > The capabilities tell the driver how to find and map the register space
> > for CXL Memory Devices. The registers are required to utilize the CXL
> > spec defined mailbox interface. The spec outlines two mailboxes, primary
> > and secondary. The secondary mailbox is earmarked for system firmware,
> > and not handled in this driver.
> >
> > Primary mailboxes are capable of generating an interrupt when submitting
> > a background command. That implementation is saved for a later time.
> >
> > Link: https://www.computeexpresslink.org/download-the-specification
> > Signed-off-by: Ben Widawsky
> > Reviewed-by: Dan Williams (v2)
>
> Looks like an off by one error in the register locator iteration.
>
> The potential buffer overrun from memcpy_fromio is still there as well
> as far as I can see.
>
> If the software provides storage for a payload of size n and the hardware
> reports a size of n + d, code will happily write beyond the end of the
> storage provided.
>
> Obviously, this shouldn't happen, but I'm not that trusting of both
> hardware and software never having bugs.
>
> Jonathan
>
> > ---
> > drivers/cxl/cxl.h | 88
> > drivers/cxl/mem.c | 543 +-
> > drivers/cxl/pci.h | 14 ++
> > 3 files changed, 643 insertions(+), 2 deletions(-)
> > create mode 100644 drivers/cxl/cxl.h
> >
> ...
>
> > +
> > +#endif /* __CXL_H__ */
> > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > index ce33c5ee77c9..b86cda2d299a 100644
> > --- a/drivers/cxl/mem.c
> > +++ b/drivers/cxl/mem.c
> > @@ -3,7 +3,458 @@
> > #include
> > #include
> > #include
> > +#include
> > #include "pci.h"
> > +#include "cxl.h"
> > +
> > +#define cxl_doorbell_busy(cxlm)
> > \
> > + (readl((cxlm)->mbox_regs + CXLDEV_MBOX_CTRL_OFFSET) & \
> > +CXLDEV_MBOX_CTRL_DOORBELL)
> > +
> > +/* CXL 2.0 - 8.2.8.4 */
> > +#define CXL_MAILBOX_TIMEOUT_MS (2 * HZ)
> > +
> > +enum opcode {
> > + CXL_MBOX_OP_IDENTIFY= 0x4000,
> > + CXL_MBOX_OP_MAX = 0x1
> > +};
> > +
> > +/**
> > + * struct mbox_cmd - A command to be submitted to hardware.
> > + * @opcode: (input) The command set and command submitted to hardware.
> > + * @payload_in: (input) Pointer to the input payload.
> > + * @payload_out: (output) Pointer to the output payload. Must be allocated
> > by
> > + * the caller.
> > + * @size_in: (input) Number of bytes to load from @payload.
> > + * @size_out: (output) Number of bytes loaded into @payload.
> > + * @return_code: (output) Error code returned from hardware.
> > + *
> > + * This is the primary mechanism used to send commands to the hardware.
> > + * All the fields except @payload_* correspond exactly to the fields
> > described in
> > + * Command Register section of the CXL 2.0 8.2.8.4.5. @payload_in and
> > + * @payload_out are written to, and read from the Command Payload Registers
> > + * defined in CXL 2.0 8.2.8.4.8.
> > + */
> > +struct mbox_cmd {
> > + u16 opcode;
> > + void *payload_in;
> > + void *payload_out;
> > + size_t size_in;
> > + size_t size_out;
> > + u16 return_code;
> > +#define CXL_MBOX_SUCCESS 0
> > +};
>
>
> > +
> > +/**
> > + * __cxl_mem_mbox_send_cmd() - Execute a mailbox command
> > + * @cxlm: The CXL memory device to communicate with.
> > + * @mbox
[PATCH v4 9/9] cxl/mem: Add payload dumping for debug
It's often useful in debug scenarios to see what the hardware has dumped
out. As it stands today, any device error will result in the payload not
being copied out, so there is no way to triage commands which weren't
expected to fail (and sometimes the payload may have that information).
The functionality is protected by normal kernel security mechanisms as
well as a CONFIG option in the CXL driver.
This was extracted from the original version of the CXL enabling patch
series.
Signed-off-by: Ben Widawsky
---
drivers/cxl/Kconfig | 13 +
drivers/cxl/mem.c | 8
2 files changed, 21 insertions(+)
diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
index 97dc4d751651..3eec9276e586 100644
--- a/drivers/cxl/Kconfig
+++ b/drivers/cxl/Kconfig
@@ -50,4 +50,17 @@ config CXL_MEM_RAW_COMMANDS
potential impact to memory currently in use by the kernel.
If developing CXL hardware or the driver say Y, otherwise say N.
+
+config CXL_MEM_INSECURE_DEBUG
+ bool "CXL.mem debugging"
+ depends on CXL_MEM
+ help
+ Enable debug of all CXL command payloads.
+
+ Some CXL devices and controllers support encryption and other
+ security features. The payloads for the commands that enable
+ those features may contain sensitive clear-text security
+ material. Disable debug of those command payloads by default.
+ If you are a kernel developer actively working on CXL
+ security enabling say Y, otherwise say N.
endif
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index dc608bb20a31..237b956f0be0 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -342,6 +342,14 @@ static int __cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
/* #5 */
rc = cxl_mem_wait_for_doorbell(cxlm);
+
+ if (!cxl_is_security_command(mbox_cmd->opcode) ||
+ IS_ENABLED(CONFIG_CXL_MEM_INSECURE_DEBUG)) {
+ print_hex_dump_debug("Payload ", DUMP_PREFIX_OFFSET, 16, 1,
+mbox_cmd->payload_in, mbox_cmd->size_in,
+true);
+ }
+
if (rc == -ETIMEDOUT) {
cxl_mem_mbox_timeout(cxlm, mbox_cmd);
return rc;
--
2.30.1
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[PATCH v4 8/9] MAINTAINERS: Add maintainers of the CXL driver
Cc: Dan Williams Cc: Vishal Verma Cc: Ira Weiny Cc: Alison Schofield Signed-off-by: Ben Widawsky --- MAINTAINERS | 11 +++ 1 file changed, 11 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 6eff4f720c72..93c8694a8f04 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -,6 +,17 @@ M: Miguel Ojeda S: Maintained F: include/linux/compiler_attributes.h +COMPUTE EXPRESS LINK (CXL) +M: Alison Schofield +M: Vishal Verma +M: Ira Weiny +M: Ben Widawsky +M: Dan Williams +L: linux-...@vger.kernel.org +S: Maintained +F: drivers/cxl/ +F: include/uapi/linux/cxl_mem.h + CONEXANT ACCESSRUNNER USB DRIVER L: accessrunner-gene...@lists.sourceforge.net S: Orphan -- 2.30.1 ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[PATCH v4 7/9] cxl/mem: Add set of informational commands
Add initial set of formal commands beyond basic identify and command
enumeration.
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams
Reviewed-by: Jonathan Cameron (v2)
---
drivers/cxl/mem.c| 9 +
include/uapi/linux/cxl_mem.h | 5 +
2 files changed, 14 insertions(+)
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 4d921b4375f9..dc608bb20a31 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -45,12 +45,16 @@
enum opcode {
CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
+ CXL_MBOX_OP_GET_FW_INFO = 0x0200,
CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
CXL_MBOX_OP_GET_LOG = 0x0401,
CXL_MBOX_OP_IDENTIFY= 0x4000,
+ CXL_MBOX_OP_GET_PARTITION_INFO = 0x4100,
CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
+ CXL_MBOX_OP_GET_LSA = 0x4102,
CXL_MBOX_OP_SET_LSA = 0x4103,
+ CXL_MBOX_OP_GET_HEALTH_INFO = 0x4200,
CXL_MBOX_OP_SET_SHUTDOWN_STATE = 0x4204,
CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
@@ -167,6 +171,11 @@ static struct cxl_mem_command mem_commands[] = {
CXL_CMD(RAW, ~0, ~0, 0),
#endif
CXL_CMD(GET_SUPPORTED_LOGS, 0, ~0, CXL_CMD_FLAG_FORCE_ENABLE),
+ CXL_CMD(GET_FW_INFO, 0, 0x50, 0),
+ CXL_CMD(GET_PARTITION_INFO, 0, 0x20, 0),
+ CXL_CMD(GET_LSA, 0x8, ~0, 0),
+ CXL_CMD(GET_HEALTH_INFO, 0, 0x12, 0),
+ CXL_CMD(GET_LOG, 0x18, ~0, CXL_CMD_FLAG_FORCE_ENABLE),
};
/*
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index 3b5bf4b58fb4..7670fe0e605a 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -24,6 +24,11 @@
___C(IDENTIFY, "Identify Command"), \
___C(RAW, "Raw device command"), \
___C(GET_SUPPORTED_LOGS, "Get Supported Logs"), \
+ ___C(GET_FW_INFO, "Get FW Info"), \
+ ___C(GET_PARTITION_INFO, "Get Partition Information"),\
+ ___C(GET_LSA, "Get Label Storage Area"), \
+ ___C(GET_HEALTH_INFO, "Get Health Info"), \
+ ___C(GET_LOG, "Get Log"), \
___C(MAX, "invalid / last command")
#define ___C(a, b) CXL_MEM_COMMAND_ID_##a
--
2.30.1
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[PATCH v4 6/9] cxl/mem: Enable commands via CEL
CXL devices identified by the memory-device class code must implement
the Device Command Interface (described in 8.2.9 of the CXL 2.0 spec).
While the driver already maintains a list of commands it supports, there
is still a need to be able to distinguish between commands that the
driver knows about from commands that are optionally supported by the
hardware.
The Command Effects Log (CEL) is specified in the CXL 2.0 specification.
The CEL is one of two types of logs, the other being vendor specific.
They are distinguished in hardware/spec via UUID. The CEL is useful for
2 things:
1. Determine which optional commands are supported by the CXL device.
2. Enumerate any vendor specific commands
The CEL is used by the driver to determine which commands are available
in the hardware and therefore which commands userspace is allowed to
execute. The set of enabled commands might be a subset of commands which
are advertised in UAPI via CXL_MEM_SEND_COMMAND IOCTL.
With the CEL enabling comes a internal flag to indicate a base set of
commands that are enabled regardless of CEL. Such commands are required
for basic interaction with the hardware and thus can be useful in debug
cases, for example if the CEL is corrupted.
The implementation leaves the statically defined table of commands and
supplements it with a bitmap to determine commands that are enabled.
This organization was chosen for the following reasons:
- Smaller memory footprint. Doesn't need a table per device.
- Reduce memory allocation complexity.
- Fixed command IDs to opcode mapping for all devices makes development
and debugging easier.
- Certain helpers are easily achievable, like cxl_for_each_cmd().
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams
---
drivers/cxl/cxl.h| 2 +
drivers/cxl/mem.c| 213 ++-
include/uapi/linux/cxl_mem.h | 1 +
3 files changed, 213 insertions(+), 3 deletions(-)
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index f69313dc3b4e..cb7a77ed443d 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -67,6 +67,7 @@ struct cxl_memdev;
*(CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
* @mbox_mutex: Mutex to synchronize mailbox access.
* @firmware_version: Firmware version for the memory device.
+ * @enabled_commands: Hardware commands found enabled in CEL.
* @pmem: Persistent memory capacity information.
* @ram: Volatile memory capacity information.
*/
@@ -82,6 +83,7 @@ struct cxl_mem {
size_t payload_size;
struct mutex mbox_mutex; /* Protects device mailbox and firmware */
char firmware_version[0x10];
+ unsigned long *enabled_cmds;
struct range pmem_range;
struct range ram_range;
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 6b4feb0ce47d..4d921b4375f9 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -46,6 +46,8 @@ enum opcode {
CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
+ CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
+ CXL_MBOX_OP_GET_LOG = 0x0401,
CXL_MBOX_OP_IDENTIFY= 0x4000,
CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
CXL_MBOX_OP_SET_LSA = 0x4103,
@@ -104,10 +106,28 @@ static DEFINE_IDA(cxl_memdev_ida);
static struct dentry *cxl_debugfs;
static bool cxl_raw_allow_all;
+enum {
+ CEL_UUID,
+ VENDOR_DEBUG_UUID,
+};
+
+/* See CXL 2.0 Table 170. Get Log Input Payload */
+static const uuid_t log_uuid[] = {
+ [CEL_UUID] = UUID_INIT(0xda9c0b5, 0xbf41, 0x4b78, 0x8f, 0x79, 0x96,
+ 0xb1, 0x62, 0x3b, 0x3f, 0x17),
+ [VENDOR_DEBUG_UUID] = UUID_INIT(0xe1819d9, 0x11a9, 0x400c, 0x81, 0x1f,
+ 0xd6, 0x07, 0x19, 0x40, 0x3d, 0x86),
+};
+
/**
* struct cxl_mem_command - Driver representation of a memory device command
* @info: Command information as it exists for the UAPI
* @opcode: The actual bits used for the mailbox protocol
+ * @flags: Set of flags effecting driver behavior.
+ *
+ * * %CXL_CMD_FLAG_FORCE_ENABLE: In cases of error, commands with this flag
+ *will be enabled by the driver regardless of what hardware may have
+ *advertised.
*
* The cxl_mem_command is the driver's internal representation of commands that
* are supported by the driver. Some of these commands may not be supported by
@@ -119,9 +139,12 @@ static bool cxl_raw_allow_all;
struct cxl_mem_command {
struct cxl_command_info info;
enum opcode opcode;
+ u32 flags;
+#define CXL_CMD_FLAG_NONE 0
+#define CXL_CMD_FLAG_FORCE_ENABLE BIT(0)
};
-#define CXL_CMD(_id, sin, sout)
\
+#define CXL_CMD(_id, sin, sout, _flags)
\
[CXL_MEM_COMMAN
[PATCH v4 5/9] cxl/mem: Add a "RAW" send command
The CXL memory device send interface will have a number of supported
commands. The raw command is not such a command. Raw commands allow
userspace to send a specified opcode to the underlying hardware and
bypass all driver checks on the command. The primary use for this
command is to [begrudgingly] allow undocumented vendor specific hardware
commands.
While not the main motivation, it also allows prototyping new hardware
commands without a driver patch and rebuild.
While this all sounds very powerful it comes with a couple of caveats:
1. Bug reports using raw commands will not get the same level of
attention as bug reports using supported commands (via taint).
2. Supported commands will be rejected by the RAW command.
With this comes new debugfs knob to allow full access to your toes with
your weapon of choice.
Cc: Ariel Sibley
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams (v2)
---
drivers/cxl/Kconfig | 18 +
drivers/cxl/mem.c| 132 +++
include/uapi/linux/cxl_mem.h | 12 +++-
3 files changed, 161 insertions(+), 1 deletion(-)
diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
index 9e80b311e928..97dc4d751651 100644
--- a/drivers/cxl/Kconfig
+++ b/drivers/cxl/Kconfig
@@ -32,4 +32,22 @@ config CXL_MEM
Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification.
If unsure say 'm'.
+
+config CXL_MEM_RAW_COMMANDS
+ bool "RAW Command Interface for Memory Devices"
+ depends on CXL_MEM
+ help
+ Enable CXL RAW command interface.
+
+ The CXL driver ioctl interface may assign a kernel ioctl command
+ number for each specification defined opcode. At any given point in
+ time the number of opcodes that the specification defines and a device
+ may implement may exceed the kernel's set of associated ioctl function
+ numbers. The mismatch is either by omission, specification is too new,
+ or by design. When prototyping new hardware, or developing / debugging
+ the driver it is useful to be able to submit any possible command to
+ the hardware, even commands that may crash the kernel due to their
+ potential impact to memory currently in use by the kernel.
+
+ If developing CXL hardware or the driver say Y, otherwise say N.
endif
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index a4298cb1182d..6b4feb0ce47d 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1,6 +1,8 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
#include
+#include
+#include
#include
#include
#include
@@ -42,7 +44,14 @@
enum opcode {
CXL_MBOX_OP_INVALID = 0x,
+ CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
+ CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
CXL_MBOX_OP_IDENTIFY= 0x4000,
+ CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
+ CXL_MBOX_OP_SET_LSA = 0x4103,
+ CXL_MBOX_OP_SET_SHUTDOWN_STATE = 0x4204,
+ CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
+ CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
CXL_MBOX_OP_MAX = 0x1
};
@@ -92,6 +101,8 @@ struct cxl_memdev {
static int cxl_mem_major;
static DEFINE_IDA(cxl_memdev_ida);
+static struct dentry *cxl_debugfs;
+static bool cxl_raw_allow_all;
/**
* struct cxl_mem_command - Driver representation of a memory device command
@@ -128,6 +139,49 @@ struct cxl_mem_command {
*/
static struct cxl_mem_command mem_commands[] = {
CXL_CMD(IDENTIFY, 0, 0x43),
+#ifdef CONFIG_CXL_MEM_RAW_COMMANDS
+ CXL_CMD(RAW, ~0, ~0),
+#endif
+};
+
+/*
+ * Commands that RAW doesn't permit. The rationale for each:
+ *
+ * CXL_MBOX_OP_ACTIVATE_FW: Firmware activation requires adjustment /
+ * coordination of transaction timeout values at the root bridge level.
+ *
+ * CXL_MBOX_OP_SET_PARTITION_INFO: The device memory map may change live
+ * and needs to be coordinated with HDM updates.
+ *
+ * CXL_MBOX_OP_SET_LSA: The label storage area may be cached by the
+ * driver and any writes from userspace invalidates those contents.
+ *
+ * CXL_MBOX_OP_SET_SHUTDOWN_STATE: Set shutdown state assumes no writes
+ * to the device after it is marked clean, userspace can not make that
+ * assertion.
+ *
+ * CXL_MBOX_OP_[GET_]SCAN_MEDIA: The kernel provides a native error list that
+ * is kept up to date with patrol notifications and error management.
+ */
+static u16 cxl_disabled_raw_commands[] = {
+ CXL_MBOX_OP_ACTIVATE_FW,
+ CXL_MBOX_OP_SET_PARTITION_INFO,
+ CXL_MBOX_OP_SET_LSA,
+ CXL_MBOX_OP_SET_SHUTDOWN_STATE,
+ CXL_MBOX_OP_SCAN_MEDIA,
+ CXL_MBOX_OP_GET_SCAN_MEDIA,
+};
+
+/*
+ * Command sets that RAW doesn't permit. All opcodes in this set are
+ * disabled because they pass plain text security payloads over the
+ * user/kernel boundary. This func
[PATCH v4 3/9] cxl/mem: Register CXL memX devices
From: Dan Williams
Create the /sys/bus/cxl hierarchy to enumerate:
* Memory Devices (per-endpoint control devices)
* Memory Address Space Devices (platform address ranges with
interleaving, performance, and persistence attributes)
* Memory Regions (active provisioned memory from an address space device
that is in use as System RAM or delegated to libnvdimm as Persistent
Memory regions).
For now, only the per-endpoint control devices are registered on the
'cxl' bus. However, going forward it will provide a mechanism to
coordinate cross-device interleave.
Signed-off-by: Dan Williams
Signed-off-by: Ben Widawsky
Reviewed-by: Jonathan Cameron (v2)
---
Documentation/ABI/testing/sysfs-bus-cxl | 26 ++
.../driver-api/cxl/memory-devices.rst | 17 +
drivers/cxl/Makefile | 3 +
drivers/cxl/bus.c | 29 ++
drivers/cxl/cxl.h | 3 +
drivers/cxl/mem.c | 301 +-
6 files changed, 377 insertions(+), 2 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl
create mode 100644 drivers/cxl/bus.c
diff --git a/Documentation/ABI/testing/sysfs-bus-cxl
b/Documentation/ABI/testing/sysfs-bus-cxl
new file mode 100644
index ..2fe7490ad6a8
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-bus-cxl
@@ -0,0 +1,26 @@
+What: /sys/bus/cxl/devices/memX/firmware_version
+Date: December, 2020
+KernelVersion: v5.12
+Contact: linux-...@vger.kernel.org
+Description:
+ (RO) "FW Revision" string as reported by the Identify
+ Memory Device Output Payload in the CXL-2.0
+ specification.
+
+What: /sys/bus/cxl/devices/memX/ram/size
+Date: December, 2020
+KernelVersion: v5.12
+Contact: linux-...@vger.kernel.org
+Description:
+ (RO) "Volatile Only Capacity" as bytes. Represents the
+ identically named field in the Identify Memory Device Output
+ Payload in the CXL-2.0 specification.
+
+What: /sys/bus/cxl/devices/memX/pmem/size
+Date: December, 2020
+KernelVersion: v5.12
+Contact: linux-...@vger.kernel.org
+Description:
+ (RO) "Persistent Only Capacity" as bytes. Represents the
+ identically named field in the Identify Memory Device Output
+ Payload in the CXL-2.0 specification.
diff --git a/Documentation/driver-api/cxl/memory-devices.rst
b/Documentation/driver-api/cxl/memory-devices.rst
index 43177e700d62..1bad466f9167 100644
--- a/Documentation/driver-api/cxl/memory-devices.rst
+++ b/Documentation/driver-api/cxl/memory-devices.rst
@@ -27,3 +27,20 @@ CXL Memory Device
.. kernel-doc:: drivers/cxl/mem.c
:internal:
+
+CXL Bus
+---
+.. kernel-doc:: drivers/cxl/bus.c
+ :doc: cxl bus
+
+External Interfaces
+===
+
+CXL IOCTL Interface
+---
+
+.. kernel-doc:: include/uapi/linux/cxl_mem.h
+ :doc: UAPI
+
+.. kernel-doc:: include/uapi/linux/cxl_mem.h
+ :internal:
diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile
index 4a30f7c3fc4a..a314a1891f4d 100644
--- a/drivers/cxl/Makefile
+++ b/drivers/cxl/Makefile
@@ -1,4 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_CXL_BUS) += cxl_bus.o
obj-$(CONFIG_CXL_MEM) += cxl_mem.o
+ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
+cxl_bus-y := bus.o
cxl_mem-y := mem.o
diff --git a/drivers/cxl/bus.c b/drivers/cxl/bus.c
new file mode 100644
index ..58f74796d525
--- /dev/null
+++ b/drivers/cxl/bus.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include
+#include
+
+/**
+ * DOC: cxl bus
+ *
+ * The CXL bus provides namespace for control devices and a rendezvous
+ * point for cross-device interleave coordination.
+ */
+struct bus_type cxl_bus_type = {
+ .name = "cxl",
+};
+EXPORT_SYMBOL_GPL(cxl_bus_type);
+
+static __init int cxl_bus_init(void)
+{
+ return bus_register(&cxl_bus_type);
+}
+
+static void cxl_bus_exit(void)
+{
+ bus_unregister(&cxl_bus_type);
+}
+
+module_init(cxl_bus_init);
+module_exit(cxl_bus_exit);
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 0806e6bb8a90..f69313dc3b4e 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -55,6 +55,7 @@
(FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \
CXLMDEV_RESET_NEEDED_NOT)
+struct cxl_memdev;
/**
* struct cxl_mem - A CXL memory device
* @pdev: The PCI device associated with this CXL device.
@@ -72,6 +73,7 @@
struct cxl_mem {
struct pci_dev *pdev;
void __iomem *regs;
+ struct cxl_memdev *cxlmd;
void __iomem *status_regs;
void __iomem *mbox_regs;
@@ -85,4 +87,5 @@ struct cxl_mem {
struct range ram_range;
};
+exter
[PATCH v4 4/9] cxl/mem: Add basic IOCTL interface
Add a straightforward IOCTL that provides a mechanism for userspace to
query the supported memory device commands. CXL commands as they appear
to userspace are described as part of the UAPI kerneldoc. The command
list returned via this IOCTL will contain the full set of commands that
the driver supports, however, some of those commands may not be
available for use by userspace.
Memory device commands first appear in the CXL 2.0 specification. They
are submitted through a mailbox mechanism specified in the CXL 2.0
specification.
The send command allows userspace to issue mailbox commands directly to
the hardware. The list of available commands to send are the output of
the query command. The driver verifies basic properties of the command
and possibly inspect the input (or output) payload to determine whether
or not the command is allowed (or might taint the kernel).
Reported-by: kernel test robot # bug in earlier revision
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams (v2)
---
.clang-format | 1 +
.../userspace-api/ioctl/ioctl-number.rst | 1 +
drivers/cxl/mem.c | 288 +-
include/uapi/linux/cxl_mem.h | 154 ++
4 files changed, 443 insertions(+), 1 deletion(-)
create mode 100644 include/uapi/linux/cxl_mem.h
diff --git a/.clang-format b/.clang-format
index 10dc5a9a61b3..3f11c8901b43 100644
--- a/.clang-format
+++ b/.clang-format
@@ -109,6 +109,7 @@ ForEachMacros:
- 'css_for_each_child'
- 'css_for_each_descendant_post'
- 'css_for_each_descendant_pre'
+ - 'cxl_for_each_cmd'
- 'device_for_each_child_node'
- 'dma_fence_chain_for_each'
- 'do_for_each_ftrace_op'
diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst
b/Documentation/userspace-api/ioctl/ioctl-number.rst
index a4c75a28c839..6eb8e634664d 100644
--- a/Documentation/userspace-api/ioctl/ioctl-number.rst
+++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
@@ -352,6 +352,7 @@ Code Seq#Include File
Comments
<mailto:michael.kl...@puffin.lb.shuttle.de>
0xCC 00-0F drivers/misc/ibmvmc.h pseries
VMC driver
0xCD 01 linux/reiserfs_fs.h
+0xCE 01-02 uapi/linux/cxl_mem.hCompute
Express Link Memory Devices
0xCF 02 fs/cifs/ioctl.c
0xDB 00-0F drivers/char/mwave/mwavepub.h
0xDD 00-3F ZFCP
device driver see drivers/s390/scsi/
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 410adb1bdffc..a4298cb1182d 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include
#include
#include
#include
@@ -40,6 +41,7 @@
#define CXL_MAILBOX_TIMEOUT_MS (2 * HZ)
enum opcode {
+ CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_IDENTIFY= 0x4000,
CXL_MBOX_OP_MAX = 0x1
};
@@ -91,6 +93,49 @@ struct cxl_memdev {
static int cxl_mem_major;
static DEFINE_IDA(cxl_memdev_ida);
+/**
+ * struct cxl_mem_command - Driver representation of a memory device command
+ * @info: Command information as it exists for the UAPI
+ * @opcode: The actual bits used for the mailbox protocol
+ *
+ * The cxl_mem_command is the driver's internal representation of commands that
+ * are supported by the driver. Some of these commands may not be supported by
+ * the hardware. The driver will use @info to validate the fields passed in by
+ * the user then submit the @opcode to the hardware.
+ *
+ * See struct cxl_command_info.
+ */
+struct cxl_mem_command {
+ struct cxl_command_info info;
+ enum opcode opcode;
+};
+
+#define CXL_CMD(_id, sin, sout)
\
+ [CXL_MEM_COMMAND_ID_##_id] = { \
+ .info = { \
+ .id = CXL_MEM_COMMAND_ID_##_id,\
+ .size_in = sin,\
+ .size_out = sout, \
+ }, \
+ .opcode = CXL_MBOX_OP_##_id, \
+ }
+
+/*
+ * This table defines the supported mailbox commands for the driver. This table
+ * is made up of a UAPI structure. Non-negative values as parameters in the
+ * table will be validated against the user's input. For example, if size_in is
+ * 0, and the user passed in 1, it is an error.
+ */
+static struct cxl_mem_command mem_
[PATCH v4 0/9] CXL 2.0 Support
Single type3 device -device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M -device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M,lsa=cxl-mem1-lsa --- [1]: https://lore.kernel.org/linux-cxl/2021021541.2123505-1-ben.widaw...@intel.com/ [2]: https://www.computeexpresslink.org/](https://www.computeexpresslink.org/ [3]: https://lore.kernel.org/qemu-devel/20210202005948.241655-1-ben.widaw...@intel.com/ [4]: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v4 [5]: https://github.com/pmem/ndctl/tree/cxl-2.0v2 Cc: linux-a...@vger.kernel.org Cc: linux-ker...@vger.kernel.org Cc: linux-nvdimm@lists.01.org Cc: linux-...@vger.kernel.org Cc: Bjorn Helgaas Cc: Chris Browy Cc: Christoph Hellwig Cc: Dan Williams Cc: David Hildenbrand Cc: David Rientjes Cc: Ira Weiny Cc: Jon Masters Cc: Jonathan Cameron Cc: Rafael Wysocki Cc: Randy Dunlap Cc: Vishal Verma Cc: "John Groves (jgroves)" Cc: "Kelley, Sean V" --- Ben Widawsky (7): cxl/mem: Find device capabilities cxl/mem: Add basic IOCTL interface cxl/mem: Add a "RAW" send command cxl/mem: Enable commands via CEL cxl/mem: Add set of informational commands MAINTAINERS: Add maintainers of the CXL driver cxl/mem: Add payload dumping for debug Dan Williams (2): cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints cxl/mem: Register CXL memX devices .clang-format |1 + Documentation/ABI/testing/sysfs-bus-cxl | 26 + Documentation/driver-api/cxl/index.rst| 12 + .../driver-api/cxl/memory-devices.rst | 46 + Documentation/driver-api/index.rst|1 + .../userspace-api/ioctl/ioctl-number.rst |1 + MAINTAINERS | 11 + drivers/Kconfig |1 + drivers/Makefile |1 + drivers/cxl/Kconfig | 66 + drivers/cxl/Makefile |7 + drivers/cxl/bus.c | 29 + drivers/cxl/cxl.h | 93 + drivers/cxl/mem.c | 1540 + drivers/cxl/pci.h | 31 + include/linux/pci_ids.h |1 + include/uapi/linux/cxl_mem.h | 170 ++ 17 files changed, 2037 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl create mode 100644 Documentation/driver-api/cxl/index.rst create mode 100644 Documentation/driver-api/cxl/memory-devices.rst create mode 100644 drivers/cxl/Kconfig create mode 100644 drivers/cxl/Makefile create mode 100644 drivers/cxl/bus.c create mode 100644 drivers/cxl/cxl.h create mode 100644 drivers/cxl/mem.c create mode 100644 drivers/cxl/pci.h create mode 100644 include/uapi/linux/cxl_mem.h -- 2.30.1 ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[PATCH v4 2/9] cxl/mem: Find device capabilities
Provide enough functionality to utilize the mailbox of a memory device.
The mailbox is used to interact with the firmware running on the memory
device. The flow is proven with one implemented command, "identify".
Because the class code has already told the driver this is a memory
device and the identify command is mandatory.
CXL devices contain an array of capabilities that describe the
interactions software can have with the device or firmware running on
the device. A CXL compliant device must implement the device status and
the mailbox capability. Additionally, a CXL compliant memory device must
implement the memory device capability. Each of the capabilities can
[will] provide an offset within the MMIO region for interacting with the
CXL device.
The capabilities tell the driver how to find and map the register space
for CXL Memory Devices. The registers are required to utilize the CXL
spec defined mailbox interface. The spec outlines two mailboxes, primary
and secondary. The secondary mailbox is earmarked for system firmware,
and not handled in this driver.
Primary mailboxes are capable of generating an interrupt when submitting
a background command. That implementation is saved for a later time.
Link: https://www.computeexpresslink.org/download-the-specification
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams (v2)
---
drivers/cxl/cxl.h | 88
drivers/cxl/mem.c | 543 +-
drivers/cxl/pci.h | 14 ++
3 files changed, 643 insertions(+), 2 deletions(-)
create mode 100644 drivers/cxl/cxl.h
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
new file mode 100644
index ..0806e6bb8a90
--- /dev/null
+++ b/drivers/cxl/cxl.h
@@ -0,0 +1,88 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2020 Intel Corporation. */
+
+#ifndef __CXL_H__
+#define __CXL_H__
+
+#include
+#include
+#include
+
+/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
+#define CXLDEV_CAP_ARRAY_OFFSET 0x0
+#define CXLDEV_CAP_ARRAY_CAP_ID 0
+#define CXLDEV_CAP_ARRAY_ID_MASK GENMASK_ULL(15, 0)
+#define CXLDEV_CAP_ARRAY_COUNT_MASK GENMASK_ULL(47, 32)
+/* CXL 2.0 8.2.8.2.1 CXL Device Capabilities */
+#define CXLDEV_CAP_CAP_ID_DEVICE_STATUS 0x1
+#define CXLDEV_CAP_CAP_ID_PRIMARY_MAILBOX 0x2
+#define CXLDEV_CAP_CAP_ID_SECONDARY_MAILBOX 0x3
+#define CXLDEV_CAP_CAP_ID_MEMDEV 0x4000
+
+/* CXL 2.0 8.2.8.4 Mailbox Registers */
+#define CXLDEV_MBOX_CAPS_OFFSET 0x00
+#define CXLDEV_MBOX_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0)
+#define CXLDEV_MBOX_CTRL_OFFSET 0x04
+#define CXLDEV_MBOX_CTRL_DOORBELL BIT(0)
+#define CXLDEV_MBOX_CMD_OFFSET 0x08
+#define CXLDEV_MBOX_CMD_COMMAND_OPCODE_MASK GENMASK_ULL(15, 0)
+#define CXLDEV_MBOX_CMD_PAYLOAD_LENGTH_MASK GENMASK_ULL(36, 16)
+#define CXLDEV_MBOX_STATUS_OFFSET 0x10
+#define CXLDEV_MBOX_STATUS_RET_CODE_MASK GENMASK_ULL(47, 32)
+#define CXLDEV_MBOX_BG_CMD_STATUS_OFFSET 0x18
+#define CXLDEV_MBOX_PAYLOAD_OFFSET 0x20
+
+/* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */
+#define CXLMDEV_STATUS_OFFSET 0x0
+#define CXLMDEV_DEV_FATAL BIT(0)
+#define CXLMDEV_FW_HALT BIT(1)
+#define CXLMDEV_STATUS_MEDIA_STATUS_MASK GENMASK(3, 2)
+#define CXLMDEV_MS_NOT_READY 0
+#define CXLMDEV_MS_READY 1
+#define CXLMDEV_MS_ERROR 2
+#define CXLMDEV_MS_DISABLED 3
+#define CXLMDEV_READY(status)
\
+ (FIELD_GET(CXLMDEV_STATUS_MEDIA_STATUS_MASK, status) ==\
+CXLMDEV_MS_READY)
+#define CXLMDEV_MBOX_IF_READY BIT(4)
+#define CXLMDEV_RESET_NEEDED_MASK GENMASK(7, 5)
+#define CXLMDEV_RESET_NEEDED_NOT 0
+#define CXLMDEV_RESET_NEEDED_COLD 1
+#define CXLMDEV_RESET_NEEDED_WARM 2
+#define CXLMDEV_RESET_NEEDED_HOT 3
+#define CXLMDEV_RESET_NEEDED_CXL 4
+#define CXLMDEV_RESET_NEEDED(status)
\
+ (FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \
+CXLMDEV_RESET_NEEDED_NOT)
+
+/**
+ * struct cxl_mem - A CXL memory device
+ * @pdev: The PCI device associated with this CXL device.
+ * @regs: IO mappings to the device's MMIO
+ * @status_regs: CXL 2.0 8.2.8.3 Device Status Registers
+ * @mbox_regs: CXL 2.0 8.2.8.4 Mailbox Registers
+ * @memdev_regs: CXL 2.0 8.2.8.5 Memory Device Registers
+ * @payload_size: Size of space for payload
+ *(CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
+ * @mbox_mutex: Mutex to synchronize mailbox access.
+ * @firmware_version: Firmware version for the memory device.
+ * @pmem: Persistent memory capacity information.
+ * @ram: Volatile memory capacity information.
+ */
+struct cxl_mem {
+ struct pci_dev *pdev;
+ void __iomem *regs;
+
+ void __iomem *status_regs;
+ void __iomem *mbox_regs;
+ void __iomem *memdev_regs;
+
+ size_t payload_size;
+ struct mutex mbox_mutex; /* Protects device mailbox and firmware */
+ char firmwa
[PATCH v4 1/9] cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints
From: Dan Williams The CXL.mem protocol allows a device to act as a provider of "System RAM" and/or "Persistent Memory" that is fully coherent as if the memory was attached to the typical CPU memory controller. With the CXL-2.0 specification a PCI endpoint can implement a "Type-3" device interface and give the operating system control over "Host Managed Device Memory". See section 2.3 Type 3 CXL Device. The memory range exported by the device may optionally be described by the platform firmware memory map, or by infrastructure like LIBNVDIMM to provision persistent memory capacity from one, or more, CXL.mem devices. A pre-requisite for Linux-managed memory-capacity provisioning is this cxl_mem driver that can speak the mailbox protocol defined in section 8.2.8.4 Mailbox Registers. For now just land the initial driver boiler-plate and Documentation/ infrastructure. Link: https://www.computeexpresslink.org/download-the-specification Cc: Jonathan Corbet Signed-off-by: Dan Williams Signed-off-by: Ben Widawsky Acked-by: David Rientjes (v1) Reviewed-by: Jonathan Cameron --- Documentation/driver-api/cxl/index.rst| 12 .../driver-api/cxl/memory-devices.rst | 29 + Documentation/driver-api/index.rst| 1 + drivers/Kconfig | 1 + drivers/Makefile | 1 + drivers/cxl/Kconfig | 35 +++ drivers/cxl/Makefile | 4 ++ drivers/cxl/mem.c | 62 +++ drivers/cxl/pci.h | 17 + include/linux/pci_ids.h | 1 + 10 files changed, 163 insertions(+) create mode 100644 Documentation/driver-api/cxl/index.rst create mode 100644 Documentation/driver-api/cxl/memory-devices.rst create mode 100644 drivers/cxl/Kconfig create mode 100644 drivers/cxl/Makefile create mode 100644 drivers/cxl/mem.c create mode 100644 drivers/cxl/pci.h diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst new file mode 100644 index ..036e49553542 --- /dev/null +++ b/Documentation/driver-api/cxl/index.rst @@ -0,0 +1,12 @@ +.. SPDX-License-Identifier: GPL-2.0 + + +Compute Express Link + + +.. toctree:: + :maxdepth: 1 + + memory-devices + +.. only:: subproject and html diff --git a/Documentation/driver-api/cxl/memory-devices.rst b/Documentation/driver-api/cxl/memory-devices.rst new file mode 100644 index ..43177e700d62 --- /dev/null +++ b/Documentation/driver-api/cxl/memory-devices.rst @@ -0,0 +1,29 @@ +.. SPDX-License-Identifier: GPL-2.0 +.. include:: + +=== +Compute Express Link Memory Devices +=== + +A Compute Express Link Memory Device is a CXL component that implements the +CXL.mem protocol. It contains some amount of volatile memory, persistent memory, +or both. It is enumerated as a PCI device for configuration and passing +messages over an MMIO mailbox. Its contribution to the System Physical +Address space is handled via HDM (Host Managed Device Memory) decoders +that optionally define a device's contribution to an interleaved address +range across multiple devices underneath a host-bridge or interleaved +across host-bridges. + +Driver Infrastructure += + +This section covers the driver infrastructure for a CXL memory device. + +CXL Memory Device +- + +.. kernel-doc:: drivers/cxl/mem.c + :doc: cxl mem + +.. kernel-doc:: drivers/cxl/mem.c + :internal: diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst index 2456d0a97ed8..d246a18fd78f 100644 --- a/Documentation/driver-api/index.rst +++ b/Documentation/driver-api/index.rst @@ -35,6 +35,7 @@ available subsections can be seen below. usb/index firewire pci/index + cxl/index spi i2c ipmb diff --git a/drivers/Kconfig b/drivers/Kconfig index dcecc9f6e33f..62c753a73651 100644 --- a/drivers/Kconfig +++ b/drivers/Kconfig @@ -6,6 +6,7 @@ menu "Device Drivers" source "drivers/amba/Kconfig" source "drivers/eisa/Kconfig" source "drivers/pci/Kconfig" +source "drivers/cxl/Kconfig" source "drivers/pcmcia/Kconfig" source "drivers/rapidio/Kconfig" diff --git a/drivers/Makefile b/drivers/Makefile index fd11b9ac4cc3..678ea810410f 100644 --- a/drivers/Makefile +++ b/drivers/Makefile @@ -73,6 +73,7 @@ obj-$(CONFIG_NVM) += lightnvm/ obj-y += base/ block/ misc/ mfd/ nfc/ obj-$(CONFIG_LIBNVDIMM)+= nvdimm/ obj-$(CONFIG_DAX) += dax/ +obj-$(CONFIG_CXL_BUS) += cxl/ obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/ obj-$(CONFIG_NUBUS)+= nubus/ obj-y += maci
Re: [PATCH v2 4/8] cxl/mem: Add basic IOCTL interface
On 21-02-14 16:30:09, Al Viro wrote:
> On Tue, Feb 09, 2021 at 04:02:55PM -0800, Ben Widawsky wrote:
>
> > +static int handle_mailbox_cmd_from_user(struct cxl_memdev *cxlmd,
> > + const struct cxl_mem_command *cmd,
> > + u64 in_payload, u64 out_payload,
> > + struct cxl_send_command __user *s)
> > +{
> > + struct cxl_mem *cxlm = cxlmd->cxlm;
> > + struct device *dev = &cxlmd->dev;
> > + struct mbox_cmd mbox_cmd = {
> > + .opcode = cmd->opcode,
> > + .size_in = cmd->info.size_in,
> > + };
> > + s32 user_size_out;
> > + int rc;
> > +
> > + if (get_user(user_size_out, &s->out.size))
> > + return -EFAULT;
>
> You have already copied it in. Never reread stuff from userland - it *can*
> change under you.
As it turns out, this is some leftover logic which doesn't need to exist at all,
and I'm happy to change it. Thanks for reviewing.
I wasn't familiar with this restriction though. For my edification could you
explain how that could happen? Also, is this something that should go in the
kdocs, because I don't see anything about this restriction there.
Thanks.
Ben
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH v4 06/09] cxl/mem: Enable commands via CEL
On 21-02-13 21:46:33, Ben Widawsky wrote:
> CXL devices identified by the memory-device class code must implement
> the Device Command Interface (described in 8.2.9 of the CXL 2.0 spec).
> While the driver already maintains a list of commands it supports, there
> is still a need to be able to distinguish between commands that the
> driver knows about from commands that are optionally supported by the
> hardware.
>
> The Command Effects Log (CEL) is specified in the CXL 2.0 specification.
> The CEL is one of two types of logs, the other being vendor specific.
> They are distinguished in hardware/spec via UUID. The CEL is useful for
> 2 things:
> 1. Determine which optional commands are supported by the CXL device.
> 2. Enumerate any vendor specific commands
>
> The CEL is used by the driver to determine which commands are available
> in the hardware and therefore which commands userspace is allowed to
> execute. The set of enabled commands might be a subset of commands which
> are advertised in UAPI via CXL_MEM_SEND_COMMAND IOCTL.
>
> With the CEL enabling comes a internal flag to indicate a base set of
> commands that are enabled regardless of CEL. Such commands are required
> for basic interaction with the hardware and thus can be useful in debug
> cases, for example if the CEL is corrupted.
>
> The implementation leaves the statically defined table of commands and
> supplements it with a bitmap to determine commands that are enabled.
> This organization was chosen for the following reasons:
> - Smaller memory footprint. Doesn't need a table per device.
> - Reduce memory allocation complexity.
> - Fixed command IDs to opcode mapping for all devices makes development
> and debugging easier.
> - Certain helpers are easily achievable, like cxl_for_each_cmd().
>
> Signed-off-by: Ben Widawsky
> Reviewed-by: Dan Williams
See below for explanation on why the v4...
> ---
> drivers/cxl/cxl.h| 2 +
> drivers/cxl/mem.c| 213 ++-
> include/uapi/linux/cxl_mem.h | 1 +
> 3 files changed, 213 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 63d7f7e01b83..a1aa9f787a2a 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -67,6 +67,7 @@ struct cxl_memdev;
> *(CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
> * @mbox_mutex: Mutex to synchronize mailbox access.
> * @firmware_version: Firmware version for the memory device.
> + * @enabled_commands: Hardware commands found enabled in CEL.
> * @pmem: Persistent memory capacity information.
> * @ram: Volatile memory capacity information.
> */
> @@ -82,6 +83,7 @@ struct cxl_mem {
> size_t payload_size;
> struct mutex mbox_mutex; /* Protects device mailbox and firmware */
> char firmware_version[0x10];
> + unsigned long *enabled_cmds;
>
> struct range pmem_range;
> struct range ram_range;
> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> index a819f090ffe2..712a8ccc496a 100644
> --- a/drivers/cxl/mem.c
> +++ b/drivers/cxl/mem.c
> @@ -45,6 +45,8 @@ enum opcode {
> CXL_MBOX_OP_INVALID = 0x,
> CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
> CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
> + CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
> + CXL_MBOX_OP_GET_LOG = 0x0401,
> CXL_MBOX_OP_IDENTIFY= 0x4000,
> CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
> CXL_MBOX_OP_SET_LSA = 0x4103,
> @@ -103,10 +105,28 @@ static DEFINE_IDA(cxl_memdev_ida);
> static struct dentry *cxl_debugfs;
> static bool cxl_raw_allow_all;
>
> +enum {
> + CEL_UUID,
> + VENDOR_DEBUG_UUID,
> +};
> +
> +/* See CXL 2.0 Table 170. Get Log Input Payload */
> +static const uuid_t log_uuid[] = {
> + [CEL_UUID] = UUID_INIT(0xda9c0b5, 0xbf41, 0x4b78, 0x8f, 0x79, 0x96,
> +0xb1, 0x62, 0x3b, 0x3f, 0x17),
> + [VENDOR_DEBUG_UUID] = UUID_INIT(0xe1819d9, 0x11a9, 0x400c, 0x81, 0x1f,
> + 0xd6, 0x07, 0x19, 0x40, 0x3d, 0x86),
> +};
> +
> /**
> * struct cxl_mem_command - Driver representation of a memory device command
> * @info: Command information as it exists for the UAPI
> * @opcode: The actual bits used for the mailbox protocol
> + * @flags: Set of flags effecting driver behavior.
> + *
> + * * %CXL_CMD_FLAG_FORCE_ENABLE: In cases of error, commands with this flag
> + *will be enabled by the driver regardless of what hardware may have
> + *advertised.
> *
> * The cxl_mem_command is the driver's internal representati
[PATCH v4 06/09] cxl/mem: Enable commands via CEL
CXL devices identified by the memory-device class code must implement
the Device Command Interface (described in 8.2.9 of the CXL 2.0 spec).
While the driver already maintains a list of commands it supports, there
is still a need to be able to distinguish between commands that the
driver knows about from commands that are optionally supported by the
hardware.
The Command Effects Log (CEL) is specified in the CXL 2.0 specification.
The CEL is one of two types of logs, the other being vendor specific.
They are distinguished in hardware/spec via UUID. The CEL is useful for
2 things:
1. Determine which optional commands are supported by the CXL device.
2. Enumerate any vendor specific commands
The CEL is used by the driver to determine which commands are available
in the hardware and therefore which commands userspace is allowed to
execute. The set of enabled commands might be a subset of commands which
are advertised in UAPI via CXL_MEM_SEND_COMMAND IOCTL.
With the CEL enabling comes a internal flag to indicate a base set of
commands that are enabled regardless of CEL. Such commands are required
for basic interaction with the hardware and thus can be useful in debug
cases, for example if the CEL is corrupted.
The implementation leaves the statically defined table of commands and
supplements it with a bitmap to determine commands that are enabled.
This organization was chosen for the following reasons:
- Smaller memory footprint. Doesn't need a table per device.
- Reduce memory allocation complexity.
- Fixed command IDs to opcode mapping for all devices makes development
and debugging easier.
- Certain helpers are easily achievable, like cxl_for_each_cmd().
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams
---
drivers/cxl/cxl.h| 2 +
drivers/cxl/mem.c| 213 ++-
include/uapi/linux/cxl_mem.h | 1 +
3 files changed, 213 insertions(+), 3 deletions(-)
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 63d7f7e01b83..a1aa9f787a2a 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -67,6 +67,7 @@ struct cxl_memdev;
*(CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
* @mbox_mutex: Mutex to synchronize mailbox access.
* @firmware_version: Firmware version for the memory device.
+ * @enabled_commands: Hardware commands found enabled in CEL.
* @pmem: Persistent memory capacity information.
* @ram: Volatile memory capacity information.
*/
@@ -82,6 +83,7 @@ struct cxl_mem {
size_t payload_size;
struct mutex mbox_mutex; /* Protects device mailbox and firmware */
char firmware_version[0x10];
+ unsigned long *enabled_cmds;
struct range pmem_range;
struct range ram_range;
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index a819f090ffe2..712a8ccc496a 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -45,6 +45,8 @@ enum opcode {
CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
+ CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
+ CXL_MBOX_OP_GET_LOG = 0x0401,
CXL_MBOX_OP_IDENTIFY= 0x4000,
CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
CXL_MBOX_OP_SET_LSA = 0x4103,
@@ -103,10 +105,28 @@ static DEFINE_IDA(cxl_memdev_ida);
static struct dentry *cxl_debugfs;
static bool cxl_raw_allow_all;
+enum {
+ CEL_UUID,
+ VENDOR_DEBUG_UUID,
+};
+
+/* See CXL 2.0 Table 170. Get Log Input Payload */
+static const uuid_t log_uuid[] = {
+ [CEL_UUID] = UUID_INIT(0xda9c0b5, 0xbf41, 0x4b78, 0x8f, 0x79, 0x96,
+ 0xb1, 0x62, 0x3b, 0x3f, 0x17),
+ [VENDOR_DEBUG_UUID] = UUID_INIT(0xe1819d9, 0x11a9, 0x400c, 0x81, 0x1f,
+ 0xd6, 0x07, 0x19, 0x40, 0x3d, 0x86),
+};
+
/**
* struct cxl_mem_command - Driver representation of a memory device command
* @info: Command information as it exists for the UAPI
* @opcode: The actual bits used for the mailbox protocol
+ * @flags: Set of flags effecting driver behavior.
+ *
+ * * %CXL_CMD_FLAG_FORCE_ENABLE: In cases of error, commands with this flag
+ *will be enabled by the driver regardless of what hardware may have
+ *advertised.
*
* The cxl_mem_command is the driver's internal representation of commands that
* are supported by the driver. Some of these commands may not be supported by
@@ -118,9 +138,12 @@ static bool cxl_raw_allow_all;
struct cxl_mem_command {
struct cxl_command_info info;
enum opcode opcode;
+ u32 flags;
+#define CXL_CMD_FLAG_NONE 0
+#define CXL_CMD_FLAG_FORCE_ENABLE BIT(0)
};
-#define CXL_CMD(_id, sin, sout)
\
+#define CXL_CMD(_id, sin, sout, _flags)
\
[CXL_MEM_COMMAN
[PATCH v3 8/9] MAINTAINERS: Add maintainers of the CXL driver
Cc: Dan Williams Cc: Vishal Verma Cc: Ira Weiny Cc: Alison Schofield Signed-off-by: Ben Widawsky --- MAINTAINERS | 11 +++ 1 file changed, 11 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 6eff4f720c72..93c8694a8f04 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -,6 +,17 @@ M: Miguel Ojeda S: Maintained F: include/linux/compiler_attributes.h +COMPUTE EXPRESS LINK (CXL) +M: Alison Schofield +M: Vishal Verma +M: Ira Weiny +M: Ben Widawsky +M: Dan Williams +L: linux-...@vger.kernel.org +S: Maintained +F: drivers/cxl/ +F: include/uapi/linux/cxl_mem.h + CONEXANT ACCESSRUNNER USB DRIVER L: accessrunner-gene...@lists.sourceforge.net S: Orphan -- 2.30.0 ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[RFC PATCH 9/9] cxl/mem: Add payload dumping for debug
It's often useful in debug scenarios to see what the hardware has dumped
out. As it stands today, any device error will result in the payload not
being copied out, so there is no way to triage commands which weren't
expected to fail (and sometimes the payload may have that information).
The functionality is protected by normal kernel security mechanisms as
well as a CONFIG option in the CXL driver.
This was extracted from the original version of the CXL enabling patch
series.
Signed-off-by: Ben Widawsky
---
drivers/cxl/Kconfig | 13 +
drivers/cxl/mem.c | 8
2 files changed, 21 insertions(+)
diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
index 97dc4d751651..3eec9276e586 100644
--- a/drivers/cxl/Kconfig
+++ b/drivers/cxl/Kconfig
@@ -50,4 +50,17 @@ config CXL_MEM_RAW_COMMANDS
potential impact to memory currently in use by the kernel.
If developing CXL hardware or the driver say Y, otherwise say N.
+
+config CXL_MEM_INSECURE_DEBUG
+ bool "CXL.mem debugging"
+ depends on CXL_MEM
+ help
+ Enable debug of all CXL command payloads.
+
+ Some CXL devices and controllers support encryption and other
+ security features. The payloads for the commands that enable
+ those features may contain sensitive clear-text security
+ material. Disable debug of those command payloads by default.
+ If you are a kernel developer actively working on CXL
+ security enabling say Y, otherwise say N.
endif
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 3bca8451348a..09c11935b824 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -341,6 +341,14 @@ static int __cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
/* #5 */
rc = cxl_mem_wait_for_doorbell(cxlm);
+
+ if (!cxl_is_security_command(mbox_cmd->opcode) ||
+ IS_ENABLED(CONFIG_CXL_MEM_INSECURE_DEBUG)) {
+ print_hex_dump_debug("Payload ", DUMP_PREFIX_OFFSET, 16, 1,
+mbox_cmd->payload_in, mbox_cmd->size_in,
+true);
+ }
+
if (rc == -ETIMEDOUT) {
cxl_mem_mbox_timeout(cxlm, mbox_cmd);
return rc;
--
2.30.0
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[PATCH v3 7/9] cxl/mem: Add set of informational commands
Add initial set of formal commands beyond basic identify and command
enumeration.
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams
Reviewed-by: Jonathan Cameron (v2)
---
drivers/cxl/mem.c| 9 +
include/uapi/linux/cxl_mem.h | 5 +
2 files changed, 14 insertions(+)
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index b026c1167bda..3bca8451348a 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -44,12 +44,16 @@
enum opcode {
CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
+ CXL_MBOX_OP_GET_FW_INFO = 0x0200,
CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
CXL_MBOX_OP_GET_LOG = 0x0401,
CXL_MBOX_OP_IDENTIFY= 0x4000,
+ CXL_MBOX_OP_GET_PARTITION_INFO = 0x4100,
CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
+ CXL_MBOX_OP_GET_LSA = 0x4102,
CXL_MBOX_OP_SET_LSA = 0x4103,
+ CXL_MBOX_OP_GET_HEALTH_INFO = 0x4200,
CXL_MBOX_OP_SET_SHUTDOWN_STATE = 0x4204,
CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
@@ -166,6 +170,11 @@ static struct cxl_mem_command mem_commands[] = {
CXL_CMD(RAW, ~0, ~0, 0),
#endif
CXL_CMD(GET_SUPPORTED_LOGS, 0, ~0, CXL_CMD_FLAG_FORCE_ENABLE),
+ CXL_CMD(GET_FW_INFO, 0, 0x50, 0),
+ CXL_CMD(GET_PARTITION_INFO, 0, 0x20, 0),
+ CXL_CMD(GET_LSA, 0x8, ~0, 0),
+ CXL_CMD(GET_HEALTH_INFO, 0, 0x12, 0),
+ CXL_CMD(GET_LOG, 0x18, ~0, CXL_CMD_FLAG_FORCE_ENABLE),
};
/*
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index 3b5bf4b58fb4..7670fe0e605a 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -24,6 +24,11 @@
___C(IDENTIFY, "Identify Command"), \
___C(RAW, "Raw device command"), \
___C(GET_SUPPORTED_LOGS, "Get Supported Logs"), \
+ ___C(GET_FW_INFO, "Get FW Info"), \
+ ___C(GET_PARTITION_INFO, "Get Partition Information"),\
+ ___C(GET_LSA, "Get Label Storage Area"), \
+ ___C(GET_HEALTH_INFO, "Get Health Info"), \
+ ___C(GET_LOG, "Get Log"), \
___C(MAX, "invalid / last command")
#define ___C(a, b) CXL_MEM_COMMAND_ID_##a
--
2.30.0
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[PATCH v3 6/9] cxl/mem: Enable commands via CEL
CXL devices identified by the memory-device class code must implement
the Device Command Interface (described in 8.2.9 of the CXL 2.0 spec).
While the driver already maintains a list of commands it supports, there
is still a need to be able to distinguish between commands that the
driver knows about from commands that are optionally supported by the
hardware.
The Command Effects Log (CEL) is specified in the CXL 2.0 specification.
The CEL is one of two types of logs, the other being vendor specific.
They are distinguished in hardware/spec via UUID. The CEL is useful for
2 things:
1. Determine which optional commands are supported by the CXL device.
2. Enumerate any vendor specific commands
The CEL is used by the driver to determine which commands are available
in the hardware and therefore which commands userspace is allowed to
execute. The set of enabled commands might be a subset of commands which
are advertised in UAPI via CXL_MEM_SEND_COMMAND IOCTL.
With the CEL enabling comes a internal flag to indicate a base set of
commands that are enabled regardless of CEL. Such commands are required
for basic interaction with the hardware and thus can be useful in debug
cases, for example if the CEL is corrupted.
The implementation leaves the statically defined table of commands and
supplements it with a bitmap to determine commands that are enabled.
This organization was chosen for the following reasons:
- Smaller memory footprint. Doesn't need a table per device.
- Reduce memory allocation complexity.
- Fixed command IDs to opcode mapping for all devices makes development
and debugging easier.
- Certain helpers are easily achievable, like cxl_for_each_cmd().
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams
---
drivers/cxl/cxl.h| 2 +
drivers/cxl/mem.c| 224 ++-
include/uapi/linux/cxl_mem.h | 1 +
3 files changed, 224 insertions(+), 3 deletions(-)
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 63d7f7e01b83..a1aa9f787a2a 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -67,6 +67,7 @@ struct cxl_memdev;
*(CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
* @mbox_mutex: Mutex to synchronize mailbox access.
* @firmware_version: Firmware version for the memory device.
+ * @enabled_commands: Hardware commands found enabled in CEL.
* @pmem: Persistent memory capacity information.
* @ram: Volatile memory capacity information.
*/
@@ -82,6 +83,7 @@ struct cxl_mem {
size_t payload_size;
struct mutex mbox_mutex; /* Protects device mailbox and firmware */
char firmware_version[0x10];
+ unsigned long *enabled_cmds;
struct range pmem_range;
struct range ram_range;
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index a819f090ffe2..b026c1167bda 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -45,6 +45,8 @@ enum opcode {
CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
+ CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
+ CXL_MBOX_OP_GET_LOG = 0x0401,
CXL_MBOX_OP_IDENTIFY= 0x4000,
CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
CXL_MBOX_OP_SET_LSA = 0x4103,
@@ -103,10 +105,28 @@ static DEFINE_IDA(cxl_memdev_ida);
static struct dentry *cxl_debugfs;
static bool cxl_raw_allow_all;
+enum {
+ CEL_UUID,
+ VENDOR_DEBUG_UUID,
+};
+
+/* See CXL 2.0 Table 170. Get Log Input Payload */
+static const uuid_t log_uuid[] = {
+ [CEL_UUID] = UUID_INIT(0xda9c0b5, 0xbf41, 0x4b78, 0x8f, 0x79, 0x96,
+ 0xb1, 0x62, 0x3b, 0x3f, 0x17),
+ [VENDOR_DEBUG_UUID] = UUID_INIT(0xe1819d9, 0x11a9, 0x400c, 0x81, 0x1f,
+ 0xd6, 0x07, 0x19, 0x40, 0x3d, 0x86),
+};
+
/**
* struct cxl_mem_command - Driver representation of a memory device command
* @info: Command information as it exists for the UAPI
* @opcode: The actual bits used for the mailbox protocol
+ * @flags: Set of flags effecting driver behavior.
+ *
+ * * %CXL_CMD_FLAG_FORCE_ENABLE: In cases of error, commands with this flag
+ *will be enabled by the driver regardless of what hardware may have
+ *advertised.
*
* The cxl_mem_command is the driver's internal representation of commands that
* are supported by the driver. Some of these commands may not be supported by
@@ -118,9 +138,12 @@ static bool cxl_raw_allow_all;
struct cxl_mem_command {
struct cxl_command_info info;
enum opcode opcode;
+ u32 flags;
+#define CXL_CMD_FLAG_NONE 0
+#define CXL_CMD_FLAG_FORCE_ENABLE BIT(0)
};
-#define CXL_CMD(_id, sin, sout)
\
+#define CXL_CMD(_id, sin, sout, _flags)
\
[CXL_MEM_COMMAN
[PATCH v3 5/9] cxl/mem: Add a "RAW" send command
The CXL memory device send interface will have a number of supported
commands. The raw command is not such a command. Raw commands allow
userspace to send a specified opcode to the underlying hardware and
bypass all driver checks on the command. The primary use for this
command is to [begrudgingly] allow undocumented vendor specific hardware
commands.
While not the main motivation, it also allows prototyping new hardware
commands without a driver patch and rebuild.
While this all sounds very powerful it comes with a couple of caveats:
1. Bug reports using raw commands will not get the same level of
attention as bug reports using supported commands (via taint).
2. Supported commands will be rejected by the RAW command.
With this comes new debugfs knob to allow full access to your toes with
your weapon of choice.
Cc: Ariel Sibley
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams (v2)
---
drivers/cxl/Kconfig | 18 ++
drivers/cxl/mem.c| 121 +++
include/uapi/linux/cxl_mem.h | 12 +++-
3 files changed, 150 insertions(+), 1 deletion(-)
diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
index 9e80b311e928..97dc4d751651 100644
--- a/drivers/cxl/Kconfig
+++ b/drivers/cxl/Kconfig
@@ -32,4 +32,22 @@ config CXL_MEM
Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification.
If unsure say 'm'.
+
+config CXL_MEM_RAW_COMMANDS
+ bool "RAW Command Interface for Memory Devices"
+ depends on CXL_MEM
+ help
+ Enable CXL RAW command interface.
+
+ The CXL driver ioctl interface may assign a kernel ioctl command
+ number for each specification defined opcode. At any given point in
+ time the number of opcodes that the specification defines and a device
+ may implement may exceed the kernel's set of associated ioctl function
+ numbers. The mismatch is either by omission, specification is too new,
+ or by design. When prototyping new hardware, or developing / debugging
+ the driver it is useful to be able to submit any possible command to
+ the hardware, even commands that may crash the kernel due to their
+ potential impact to memory currently in use by the kernel.
+
+ If developing CXL hardware or the driver say Y, otherwise say N.
endif
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index d764a35afea9..a819f090ffe2 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1,6 +1,8 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
#include
+#include
+#include
#include
#include
#include
@@ -41,7 +43,14 @@
enum opcode {
CXL_MBOX_OP_INVALID = 0x,
+ CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
+ CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
CXL_MBOX_OP_IDENTIFY= 0x4000,
+ CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
+ CXL_MBOX_OP_SET_LSA = 0x4103,
+ CXL_MBOX_OP_SET_SHUTDOWN_STATE = 0x4204,
+ CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
+ CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
CXL_MBOX_OP_MAX = 0x1
};
@@ -91,6 +100,8 @@ struct cxl_memdev {
static int cxl_mem_major;
static DEFINE_IDA(cxl_memdev_ida);
+static struct dentry *cxl_debugfs;
+static bool cxl_raw_allow_all;
/**
* struct cxl_mem_command - Driver representation of a memory device command
@@ -127,6 +138,49 @@ struct cxl_mem_command {
*/
static struct cxl_mem_command mem_commands[] = {
CXL_CMD(IDENTIFY, 0, 0x43),
+#ifdef CONFIG_CXL_MEM_RAW_COMMANDS
+ CXL_CMD(RAW, ~0, ~0),
+#endif
+};
+
+/*
+ * Commands that RAW doesn't permit. The rationale for each:
+ *
+ * CXL_MBOX_OP_ACTIVATE_FW: Firmware activation requires adjustment /
+ * coordination of transaction timeout values at the root bridge level.
+ *
+ * CXL_MBOX_OP_SET_PARTITION_INFO: The device memory map may change live
+ * and needs to be coordinated with HDM updates.
+ *
+ * CXL_MBOX_OP_SET_LSA: The label storage area may be cached by the
+ * driver and any writes from userspace invalidates those contents.
+ *
+ * CXL_MBOX_OP_SET_SHUTDOWN_STATE: Set shutdown state assumes no writes
+ * to the device after it is marked clean, userspace can not make that
+ * assertion.
+ *
+ * CXL_MBOX_OP_[GET_]SCAN_MEDIA: The kernel provides a native error list that
+ * is kept up to date with patrol notifications and error management.
+ */
+static u16 cxl_disabled_raw_commands[] = {
+ CXL_MBOX_OP_ACTIVATE_FW,
+ CXL_MBOX_OP_SET_PARTITION_INFO,
+ CXL_MBOX_OP_SET_LSA,
+ CXL_MBOX_OP_SET_SHUTDOWN_STATE,
+ CXL_MBOX_OP_SCAN_MEDIA,
+ CXL_MBOX_OP_GET_SCAN_MEDIA,
+};
+
+/*
+ * Command sets that RAW doesn't permit. All opcodes in this set are
+ * disabled because they pass plain text security payloads over the
+ * user/kernel boundary. T
[PATCH v3 4/9] cxl/mem: Add basic IOCTL interface
Add a straightforward IOCTL that provides a mechanism for userspace to
query the supported memory device commands. CXL commands as they appear
to userspace are described as part of the UAPI kerneldoc. The command
list returned via this IOCTL will contain the full set of commands that
the driver supports, however, some of those commands may not be
available for use by userspace.
Memory device commands first appear in the CXL 2.0 specification. They
are submitted through a mailbox mechanism specified in the CXL 2.0
specification.
The send command allows userspace to issue mailbox commands directly to
the hardware. The list of available commands to send are the output of
the query command. The driver verifies basic properties of the command
and possibly inspect the input (or output) payload to determine whether
or not the command is allowed (or might taint the kernel).
Reported-by: kernel test robot # bug in earlier revision
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams (v2)
---
.clang-format | 1 +
.../userspace-api/ioctl/ioctl-number.rst | 1 +
drivers/cxl/mem.c | 280 +-
include/uapi/linux/cxl_mem.h | 154 ++
4 files changed, 435 insertions(+), 1 deletion(-)
create mode 100644 include/uapi/linux/cxl_mem.h
diff --git a/.clang-format b/.clang-format
index 10dc5a9a61b3..3f11c8901b43 100644
--- a/.clang-format
+++ b/.clang-format
@@ -109,6 +109,7 @@ ForEachMacros:
- 'css_for_each_child'
- 'css_for_each_descendant_post'
- 'css_for_each_descendant_pre'
+ - 'cxl_for_each_cmd'
- 'device_for_each_child_node'
- 'dma_fence_chain_for_each'
- 'do_for_each_ftrace_op'
diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst
b/Documentation/userspace-api/ioctl/ioctl-number.rst
index a4c75a28c839..6eb8e634664d 100644
--- a/Documentation/userspace-api/ioctl/ioctl-number.rst
+++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
@@ -352,6 +352,7 @@ Code Seq#Include File
Comments
<mailto:michael.kl...@puffin.lb.shuttle.de>
0xCC 00-0F drivers/misc/ibmvmc.h pseries
VMC driver
0xCD 01 linux/reiserfs_fs.h
+0xCE 01-02 uapi/linux/cxl_mem.hCompute
Express Link Memory Devices
0xCF 02 fs/cifs/ioctl.c
0xDB 00-0F drivers/char/mwave/mwavepub.h
0xDD 00-3F ZFCP
device driver see drivers/s390/scsi/
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index f03482aa155d..d764a35afea9 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include
#include
#include
#include
@@ -39,6 +40,7 @@
#define CXL_MAILBOX_TIMEOUT_MS (2 * HZ)
enum opcode {
+ CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_IDENTIFY= 0x4000,
CXL_MBOX_OP_MAX = 0x1
};
@@ -90,6 +92,49 @@ struct cxl_memdev {
static int cxl_mem_major;
static DEFINE_IDA(cxl_memdev_ida);
+/**
+ * struct cxl_mem_command - Driver representation of a memory device command
+ * @info: Command information as it exists for the UAPI
+ * @opcode: The actual bits used for the mailbox protocol
+ *
+ * The cxl_mem_command is the driver's internal representation of commands that
+ * are supported by the driver. Some of these commands may not be supported by
+ * the hardware. The driver will use @info to validate the fields passed in by
+ * the user then submit the @opcode to the hardware.
+ *
+ * See struct cxl_command_info.
+ */
+struct cxl_mem_command {
+ struct cxl_command_info info;
+ enum opcode opcode;
+};
+
+#define CXL_CMD(_id, sin, sout)
\
+ [CXL_MEM_COMMAND_ID_##_id] = { \
+ .info = { \
+ .id = CXL_MEM_COMMAND_ID_##_id,\
+ .size_in = sin,\
+ .size_out = sout, \
+ }, \
+ .opcode = CXL_MBOX_OP_##_id, \
+ }
+
+/*
+ * This table defines the supported mailbox commands for the driver. This table
+ * is made up of a UAPI structure. Non-negative values as parameters in the
+ * table will be validated against the user's input. For example, if size_in is
+ * 0, and the user passed in 1, it is an error.
+ */
+static struct cxl_mem_command mem_
[PATCH v3 3/9] cxl/mem: Register CXL memX devices
From: Dan Williams
Create the /sys/bus/cxl hierarchy to enumerate:
* Memory Devices (per-endpoint control devices)
* Memory Address Space Devices (platform address ranges with
interleaving, performance, and persistence attributes)
* Memory Regions (active provisioned memory from an address space device
that is in use as System RAM or delegated to libnvdimm as Persistent
Memory regions).
For now, only the per-endpoint control devices are registered on the
'cxl' bus. However, going forward it will provide a mechanism to
coordinate cross-device interleave.
Signed-off-by: Dan Williams
Signed-off-by: Ben Widawsky
Reviewed-by: Jonathan Cameron (v2)
---
Documentation/ABI/testing/sysfs-bus-cxl | 26 ++
.../driver-api/cxl/memory-devices.rst | 17 +
drivers/cxl/Makefile | 3 +
drivers/cxl/bus.c | 29 ++
drivers/cxl/cxl.h | 3 +
drivers/cxl/mem.c | 301 +-
6 files changed, 377 insertions(+), 2 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl
create mode 100644 drivers/cxl/bus.c
diff --git a/Documentation/ABI/testing/sysfs-bus-cxl
b/Documentation/ABI/testing/sysfs-bus-cxl
new file mode 100644
index ..2fe7490ad6a8
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-bus-cxl
@@ -0,0 +1,26 @@
+What: /sys/bus/cxl/devices/memX/firmware_version
+Date: December, 2020
+KernelVersion: v5.12
+Contact: linux-...@vger.kernel.org
+Description:
+ (RO) "FW Revision" string as reported by the Identify
+ Memory Device Output Payload in the CXL-2.0
+ specification.
+
+What: /sys/bus/cxl/devices/memX/ram/size
+Date: December, 2020
+KernelVersion: v5.12
+Contact: linux-...@vger.kernel.org
+Description:
+ (RO) "Volatile Only Capacity" as bytes. Represents the
+ identically named field in the Identify Memory Device Output
+ Payload in the CXL-2.0 specification.
+
+What: /sys/bus/cxl/devices/memX/pmem/size
+Date: December, 2020
+KernelVersion: v5.12
+Contact: linux-...@vger.kernel.org
+Description:
+ (RO) "Persistent Only Capacity" as bytes. Represents the
+ identically named field in the Identify Memory Device Output
+ Payload in the CXL-2.0 specification.
diff --git a/Documentation/driver-api/cxl/memory-devices.rst
b/Documentation/driver-api/cxl/memory-devices.rst
index 43177e700d62..1bad466f9167 100644
--- a/Documentation/driver-api/cxl/memory-devices.rst
+++ b/Documentation/driver-api/cxl/memory-devices.rst
@@ -27,3 +27,20 @@ CXL Memory Device
.. kernel-doc:: drivers/cxl/mem.c
:internal:
+
+CXL Bus
+---
+.. kernel-doc:: drivers/cxl/bus.c
+ :doc: cxl bus
+
+External Interfaces
+===
+
+CXL IOCTL Interface
+---
+
+.. kernel-doc:: include/uapi/linux/cxl_mem.h
+ :doc: UAPI
+
+.. kernel-doc:: include/uapi/linux/cxl_mem.h
+ :internal:
diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile
index 4a30f7c3fc4a..a314a1891f4d 100644
--- a/drivers/cxl/Makefile
+++ b/drivers/cxl/Makefile
@@ -1,4 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_CXL_BUS) += cxl_bus.o
obj-$(CONFIG_CXL_MEM) += cxl_mem.o
+ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
+cxl_bus-y := bus.o
cxl_mem-y := mem.o
diff --git a/drivers/cxl/bus.c b/drivers/cxl/bus.c
new file mode 100644
index ..58f74796d525
--- /dev/null
+++ b/drivers/cxl/bus.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include
+#include
+
+/**
+ * DOC: cxl bus
+ *
+ * The CXL bus provides namespace for control devices and a rendezvous
+ * point for cross-device interleave coordination.
+ */
+struct bus_type cxl_bus_type = {
+ .name = "cxl",
+};
+EXPORT_SYMBOL_GPL(cxl_bus_type);
+
+static __init int cxl_bus_init(void)
+{
+ return bus_register(&cxl_bus_type);
+}
+
+static void cxl_bus_exit(void)
+{
+ bus_unregister(&cxl_bus_type);
+}
+
+module_init(cxl_bus_init);
+module_exit(cxl_bus_exit);
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 9cd9bc79fc48..63d7f7e01b83 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -55,6 +55,7 @@
(FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \
CXLMDEV_RESET_NEEDED_NOT)
+struct cxl_memdev;
/**
* struct cxl_mem - A CXL memory device
* @pdev: The PCI device associated with this CXL device.
@@ -72,6 +73,7 @@
struct cxl_mem {
struct pci_dev *pdev;
void __iomem *regs;
+ struct cxl_memdev *cxlmd;
void __iomem *status_regs;
void __iomem *mbox_regs;
@@ -85,4 +87,5 @@ struct cxl_mem {
struct range ram_range;
};
+exter
[PATCH v3 2/9] cxl/mem: Find device capabilities
Provide enough functionality to utilize the mailbox of a memory device.
The mailbox is used to interact with the firmware running on the memory
device. The flow is proven with one implemented command, "identify".
Because the class code has already told the driver this is a memory
device and the identify command is mandatory.
CXL devices contain an array of capabilities that describe the
interactions software can have with the device or firmware running on
the device. A CXL compliant device must implement the device status and
the mailbox capability. Additionally, a CXL compliant memory device must
implement the memory device capability. Each of the capabilities can
[will] provide an offset within the MMIO region for interacting with the
CXL device.
The capabilities tell the driver how to find and map the register space
for CXL Memory Devices. The registers are required to utilize the CXL
spec defined mailbox interface. The spec outlines two mailboxes, primary
and secondary. The secondary mailbox is earmarked for system firmware,
and not handled in this driver.
Primary mailboxes are capable of generating an interrupt when submitting
a background command. That implementation is saved for a later time.
Link: https://www.computeexpresslink.org/download-the-specification
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams (v2)
---
drivers/cxl/cxl.h | 88
drivers/cxl/mem.c | 542 +-
drivers/cxl/pci.h | 14 ++
3 files changed, 642 insertions(+), 2 deletions(-)
create mode 100644 drivers/cxl/cxl.h
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
new file mode 100644
index ..9cd9bc79fc48
--- /dev/null
+++ b/drivers/cxl/cxl.h
@@ -0,0 +1,88 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2020 Intel Corporation. */
+
+#ifndef __CXL_H__
+#define __CXL_H__
+
+#include
+#include
+#include
+
+/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
+#define CXLDEV_CAP_ARRAY_OFFSET 0x0
+#define CXLDEV_CAP_ARRAY_CAP_ID 0
+#define CXLDEV_CAP_ARRAY_ID_MASK GENMASK(15, 0)
+#define CXLDEV_CAP_ARRAY_COUNT_MASK GENMASK(47, 32)
+/* CXL 2.0 8.2.8.2.1 CXL Device Capabilities */
+#define CXLDEV_CAP_CAP_ID_DEVICE_STATUS 0x1
+#define CXLDEV_CAP_CAP_ID_PRIMARY_MAILBOX 0x2
+#define CXLDEV_CAP_CAP_ID_SECONDARY_MAILBOX 0x3
+#define CXLDEV_CAP_CAP_ID_MEMDEV 0x4000
+
+/* CXL 2.0 8.2.8.4 Mailbox Registers */
+#define CXLDEV_MBOX_CAPS_OFFSET 0x00
+#define CXLDEV_MBOX_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0)
+#define CXLDEV_MBOX_CTRL_OFFSET 0x04
+#define CXLDEV_MBOX_CTRL_DOORBELL BIT(0)
+#define CXLDEV_MBOX_CMD_OFFSET 0x08
+#define CXLDEV_MBOX_CMD_COMMAND_OPCODE_MASK GENMASK(15, 0)
+#define CXLDEV_MBOX_CMD_PAYLOAD_LENGTH_MASK GENMASK(36, 16)
+#define CXLDEV_MBOX_STATUS_OFFSET 0x10
+#define CXLDEV_MBOX_STATUS_RET_CODE_MASK GENMASK(47, 32)
+#define CXLDEV_MBOX_BG_CMD_STATUS_OFFSET 0x18
+#define CXLDEV_MBOX_PAYLOAD_OFFSET 0x20
+
+/* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */
+#define CXLMDEV_STATUS_OFFSET 0x0
+#define CXLMDEV_DEV_FATAL BIT(0)
+#define CXLMDEV_FW_HALT BIT(1)
+#define CXLMDEV_STATUS_MEDIA_STATUS_MASK GENMASK(3, 2)
+#define CXLMDEV_MS_NOT_READY 0
+#define CXLMDEV_MS_READY 1
+#define CXLMDEV_MS_ERROR 2
+#define CXLMDEV_MS_DISABLED 3
+#define CXLMDEV_READY(status)
\
+ (FIELD_GET(CXLMDEV_STATUS_MEDIA_STATUS_MASK, status) ==\
+CXLMDEV_MS_READY)
+#define CXLMDEV_MBOX_IF_READY BIT(4)
+#define CXLMDEV_RESET_NEEDED_MASK GENMASK(7, 5)
+#define CXLMDEV_RESET_NEEDED_NOT 0
+#define CXLMDEV_RESET_NEEDED_COLD 1
+#define CXLMDEV_RESET_NEEDED_WARM 2
+#define CXLMDEV_RESET_NEEDED_HOT 3
+#define CXLMDEV_RESET_NEEDED_CXL 4
+#define CXLMDEV_RESET_NEEDED(status)
\
+ (FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \
+CXLMDEV_RESET_NEEDED_NOT)
+
+/**
+ * struct cxl_mem - A CXL memory device
+ * @pdev: The PCI device associated with this CXL device.
+ * @regs: IO mappings to the device's MMIO
+ * @status_regs: CXL 2.0 8.2.8.3 Device Status Registers
+ * @mbox_regs: CXL 2.0 8.2.8.4 Mailbox Registers
+ * @memdev_regs: CXL 2.0 8.2.8.5 Memory Device Registers
+ * @payload_size: Size of space for payload
+ *(CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
+ * @mbox_mutex: Mutex to synchronize mailbox access.
+ * @firmware_version: Firmware version for the memory device.
+ * @pmem: Persistent memory capacity information.
+ * @ram: Volatile memory capacity information.
+ */
+struct cxl_mem {
+ struct pci_dev *pdev;
+ void __iomem *regs;
+
+ void __iomem *status_regs;
+ void __iomem *mbox_regs;
+ void __iomem *memdev_regs;
+
+ size_t payload_size;
+ struct mutex mbox_mutex; /* Protects device mailbox and firmware */
+ char firmware_version[0x10];
+
+ struct r
[PATCH v3 1/9] cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints
From: Dan Williams The CXL.mem protocol allows a device to act as a provider of "System RAM" and/or "Persistent Memory" that is fully coherent as if the memory was attached to the typical CPU memory controller. With the CXL-2.0 specification a PCI endpoint can implement a "Type-3" device interface and give the operating system control over "Host Managed Device Memory". See section 2.3 Type 3 CXL Device. The memory range exported by the device may optionally be described by the platform firmware memory map, or by infrastructure like LIBNVDIMM to provision persistent memory capacity from one, or more, CXL.mem devices. A pre-requisite for Linux-managed memory-capacity provisioning is this cxl_mem driver that can speak the mailbox protocol defined in section 8.2.8.4 Mailbox Registers. For now just land the initial driver boiler-plate and Documentation/ infrastructure. Link: https://www.computeexpresslink.org/download-the-specification Cc: Jonathan Corbet Signed-off-by: Dan Williams Signed-off-by: Ben Widawsky Acked-by: David Rientjes (v1) Reviewed-by: Jonathan Cameron --- Documentation/driver-api/cxl/index.rst| 12 .../driver-api/cxl/memory-devices.rst | 29 + Documentation/driver-api/index.rst| 1 + drivers/Kconfig | 1 + drivers/Makefile | 1 + drivers/cxl/Kconfig | 35 +++ drivers/cxl/Makefile | 4 ++ drivers/cxl/mem.c | 62 +++ drivers/cxl/pci.h | 17 + include/linux/pci_ids.h | 1 + 10 files changed, 163 insertions(+) create mode 100644 Documentation/driver-api/cxl/index.rst create mode 100644 Documentation/driver-api/cxl/memory-devices.rst create mode 100644 drivers/cxl/Kconfig create mode 100644 drivers/cxl/Makefile create mode 100644 drivers/cxl/mem.c create mode 100644 drivers/cxl/pci.h diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst new file mode 100644 index ..036e49553542 --- /dev/null +++ b/Documentation/driver-api/cxl/index.rst @@ -0,0 +1,12 @@ +.. SPDX-License-Identifier: GPL-2.0 + + +Compute Express Link + + +.. toctree:: + :maxdepth: 1 + + memory-devices + +.. only:: subproject and html diff --git a/Documentation/driver-api/cxl/memory-devices.rst b/Documentation/driver-api/cxl/memory-devices.rst new file mode 100644 index ..43177e700d62 --- /dev/null +++ b/Documentation/driver-api/cxl/memory-devices.rst @@ -0,0 +1,29 @@ +.. SPDX-License-Identifier: GPL-2.0 +.. include:: + +=== +Compute Express Link Memory Devices +=== + +A Compute Express Link Memory Device is a CXL component that implements the +CXL.mem protocol. It contains some amount of volatile memory, persistent memory, +or both. It is enumerated as a PCI device for configuration and passing +messages over an MMIO mailbox. Its contribution to the System Physical +Address space is handled via HDM (Host Managed Device Memory) decoders +that optionally define a device's contribution to an interleaved address +range across multiple devices underneath a host-bridge or interleaved +across host-bridges. + +Driver Infrastructure += + +This section covers the driver infrastructure for a CXL memory device. + +CXL Memory Device +- + +.. kernel-doc:: drivers/cxl/mem.c + :doc: cxl mem + +.. kernel-doc:: drivers/cxl/mem.c + :internal: diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst index 2456d0a97ed8..d246a18fd78f 100644 --- a/Documentation/driver-api/index.rst +++ b/Documentation/driver-api/index.rst @@ -35,6 +35,7 @@ available subsections can be seen below. usb/index firewire pci/index + cxl/index spi i2c ipmb diff --git a/drivers/Kconfig b/drivers/Kconfig index dcecc9f6e33f..62c753a73651 100644 --- a/drivers/Kconfig +++ b/drivers/Kconfig @@ -6,6 +6,7 @@ menu "Device Drivers" source "drivers/amba/Kconfig" source "drivers/eisa/Kconfig" source "drivers/pci/Kconfig" +source "drivers/cxl/Kconfig" source "drivers/pcmcia/Kconfig" source "drivers/rapidio/Kconfig" diff --git a/drivers/Makefile b/drivers/Makefile index fd11b9ac4cc3..678ea810410f 100644 --- a/drivers/Makefile +++ b/drivers/Makefile @@ -73,6 +73,7 @@ obj-$(CONFIG_NVM) += lightnvm/ obj-y += base/ block/ misc/ mfd/ nfc/ obj-$(CONFIG_LIBNVDIMM)+= nvdimm/ obj-$(CONFIG_DAX) += dax/ +obj-$(CONFIG_CXL_BUS) += cxl/ obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/ obj-$(CONFIG_NUBUS)+= nubus/ obj-y += maci
[PATCH v3 0/9] CXL 2.0 Support
| |CXL RP | | | +--+---+--+ v |+--+ || "window" | |+--+ v ^ +-+ | | CXL Type 3 | | | Device|<--+ +-+ // Memory backend for "window" -object memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M // Memory backend for LSA -object memory-backend-file,id=cxl-mem1-lsa,share,mem-path=cxl-mem1-lsa,size=1K // Host Bridge -device pxb-cxl id=cxl.0,bus=pcie.0,bus_nr=52,uid=0 len-window-base=1,window-base[0]=0x4c000 memdev[0]=cxl-mem1 // Single root port -device cxl rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0,memdev=cxl-mem1 // Single type3 device -device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M -device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M,lsa=cxl-mem1-lsa --- [1]: https://lore.kernel.org/linux-cxl/2021021259.635748-1-ben.widaw...@intel.com/ [2]: https://www.computeexpresslink.org/](https://www.computeexpresslink.org/) [3]: https://lore.kernel.org/qemu-devel/20210202005948.241655-1-ben.widaw...@intel.com/ [4]: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v4 [5]: https://github.com/pmem/ndctl/tree/cxl-2.0v2 Ben Widawsky (7): cxl/mem: Find device capabilities cxl/mem: Add basic IOCTL interface cxl/mem: Add a "RAW" send command cxl/mem: Enable commands via CEL cxl/mem: Add set of informational commands MAINTAINERS: Add maintainers of the CXL driver cxl/mem: Add payload dumping for debug Dan Williams (2): cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints cxl/mem: Register CXL memX devices .clang-format |1 + Documentation/ABI/testing/sysfs-bus-cxl | 26 + Documentation/driver-api/cxl/index.rst| 12 + .../driver-api/cxl/memory-devices.rst | 46 + Documentation/driver-api/index.rst|1 + .../userspace-api/ioctl/ioctl-number.rst |1 + MAINTAINERS | 11 + drivers/Kconfig |1 + drivers/Makefile |1 + drivers/cxl/Kconfig | 66 + drivers/cxl/Makefile |7 + drivers/cxl/bus.c | 29 + drivers/cxl/cxl.h | 93 + drivers/cxl/mem.c | 1531 + drivers/cxl/pci.h | 31 + include/linux/pci_ids.h |1 + include/uapi/linux/cxl_mem.h | 170 ++ 17 files changed, 2028 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl create mode 100644 Documentation/driver-api/cxl/index.rst create mode 100644 Documentation/driver-api/cxl/memory-devices.rst create mode 100644 drivers/cxl/Kconfig create mode 100644 drivers/cxl/Makefile create mode 100644 drivers/cxl/bus.c create mode 100644 drivers/cxl/cxl.h create mode 100644 drivers/cxl/mem.c create mode 100644 drivers/cxl/pci.h create mode 100644 include/uapi/linux/cxl_mem.h --- Cc: linux-a...@vger.kernel.org Cc: linux-ker...@vger.kernel.org Cc: linux-nvdimm@lists.01.org Cc: linux-...@vger.kernel.org Cc: Bjorn Helgaas Cc: Chris Browy Cc: Christoph Hellwig Cc: Dan Williams Cc: David Hildenbrand Cc: David Rientjes Cc: Ira Weiny Cc: Jon Masters Cc: Jonathan Cameron Cc: Rafael Wysocki Cc: Randy Dunlap Cc: Vishal Verma Cc: "John Groves (jgroves)" Cc: "Kelley, Sean V" -- 2.30.0 ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH v2 2/8] cxl/mem: Find device capabilities
On 21-02-12 13:27:06, Jonathan Cameron wrote: > On Thu, 11 Feb 2021 07:55:29 -0800 > Ben Widawsky wrote: > > > On 21-02-11 09:55:48, Jonathan Cameron wrote: > > > On Wed, 10 Feb 2021 10:16:05 -0800 > > > Ben Widawsky wrote: > > > > > > > On 21-02-10 08:55:57, Ben Widawsky wrote: > > > > > On 21-02-10 15:07:59, Jonathan Cameron wrote: > > > > > > On Wed, 10 Feb 2021 13:32:52 + > > > > > > Jonathan Cameron wrote: > > > > > > > > > > > > > On Tue, 9 Feb 2021 16:02:53 -0800 > > > > > > > Ben Widawsky wrote: > > > > > > > > > > > > > > > Provide enough functionality to utilize the mailbox of a memory > > > > > > > > device. > > > > > > > > The mailbox is used to interact with the firmware running on > > > > > > > > the memory > > > > > > > > device. The flow is proven with one implemented command, > > > > > > > > "identify". > > > > > > > > Because the class code has already told the driver this is a > > > > > > > > memory > > > > > > > > device and the identify command is mandatory. > > > > > > > > > > > > > > > > CXL devices contain an array of capabilities that describe the > > > > > > > > interactions software can have with the device or firmware > > > > > > > > running on > > > > > > > > the device. A CXL compliant device must implement the device > > > > > > > > status and > > > > > > > > the mailbox capability. Additionally, a CXL compliant memory > > > > > > > > device must > > > > > > > > implement the memory device capability. Each of the > > > > > > > > capabilities can > > > > > > > > [will] provide an offset within the MMIO region for interacting > > > > > > > > with the > > > > > > > > CXL device. > > > > > > > > > > > > > > > > The capabilities tell the driver how to find and map the > > > > > > > > register space > > > > > > > > for CXL Memory Devices. The registers are required to utilize > > > > > > > > the CXL > > > > > > > > spec defined mailbox interface. The spec outlines two > > > > > > > > mailboxes, primary > > > > > > > > and secondary. The secondary mailbox is earmarked for system > > > > > > > > firmware, > > > > > > > > and not handled in this driver. > > > > > > > > > > > > > > > > Primary mailboxes are capable of generating an interrupt when > > > > > > > > submitting > > > > > > > > a background command. That implementation is saved for a later > > > > > > > > time. > > > > > > > > > > > > > > > > Link: > > > > > > > > https://www.computeexpresslink.org/download-the-specification > > > > > > > > Signed-off-by: Ben Widawsky > > > > > > > > Reviewed-by: Dan Williams > > > > > > > > > > > > > > Hi Ben, > > > > > > > > > > > > > > > > > > > > > > +/** > > > > > > > > + * cxl_mem_mbox_send_cmd() - Send a mailbox command to a > > > > > > > > memory device. > > > > > > > > + * @cxlm: The CXL memory device to communicate with. > > > > > > > > + * @mbox_cmd: Command to send to the memory device. > > > > > > > > + * > > > > > > > > + * Context: Any context. Expects mbox_lock to be held. > > > > > > > > + * Return: -ETIMEDOUT if timeout occurred waiting for > > > > > > > > completion. 0 on success. > > > > > > > > + * Caller should check the return code in @mbox_cmd to > > > > > > > > make sure it > > > > > > > > + * succeeded. > > > > > > > > > > > > > > cxl_xfer_log() doesn't check mbox_cmd->return_co
Re: [PATCH v2 2/8] cxl/mem: Find device capabilities
On 21-02-11 09:55:48, Jonathan Cameron wrote: > On Wed, 10 Feb 2021 10:16:05 -0800 > Ben Widawsky wrote: > > > On 21-02-10 08:55:57, Ben Widawsky wrote: > > > On 21-02-10 15:07:59, Jonathan Cameron wrote: > > > > On Wed, 10 Feb 2021 13:32:52 + > > > > Jonathan Cameron wrote: > > > > > > > > > On Tue, 9 Feb 2021 16:02:53 -0800 > > > > > Ben Widawsky wrote: > > > > > > > > > > > Provide enough functionality to utilize the mailbox of a memory > > > > > > device. > > > > > > The mailbox is used to interact with the firmware running on the > > > > > > memory > > > > > > device. The flow is proven with one implemented command, "identify". > > > > > > Because the class code has already told the driver this is a memory > > > > > > device and the identify command is mandatory. > > > > > > > > > > > > CXL devices contain an array of capabilities that describe the > > > > > > interactions software can have with the device or firmware running > > > > > > on > > > > > > the device. A CXL compliant device must implement the device status > > > > > > and > > > > > > the mailbox capability. Additionally, a CXL compliant memory device > > > > > > must > > > > > > implement the memory device capability. Each of the capabilities can > > > > > > [will] provide an offset within the MMIO region for interacting > > > > > > with the > > > > > > CXL device. > > > > > > > > > > > > The capabilities tell the driver how to find and map the register > > > > > > space > > > > > > for CXL Memory Devices. The registers are required to utilize the > > > > > > CXL > > > > > > spec defined mailbox interface. The spec outlines two mailboxes, > > > > > > primary > > > > > > and secondary. The secondary mailbox is earmarked for system > > > > > > firmware, > > > > > > and not handled in this driver. > > > > > > > > > > > > Primary mailboxes are capable of generating an interrupt when > > > > > > submitting > > > > > > a background command. That implementation is saved for a later time. > > > > > > > > > > > > Link: https://www.computeexpresslink.org/download-the-specification > > > > > > Signed-off-by: Ben Widawsky > > > > > > Reviewed-by: Dan Williams > > > > > > > > > > Hi Ben, > > > > > > > > > > > > > > > > +/** > > > > > > + * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory > > > > > > device. > > > > > > + * @cxlm: The CXL memory device to communicate with. > > > > > > + * @mbox_cmd: Command to send to the memory device. > > > > > > + * > > > > > > + * Context: Any context. Expects mbox_lock to be held. > > > > > > + * Return: -ETIMEDOUT if timeout occurred waiting for completion. > > > > > > 0 on success. > > > > > > + * Caller should check the return code in @mbox_cmd to > > > > > > make sure it > > > > > > + * succeeded. > > > > > > > > > > cxl_xfer_log() doesn't check mbox_cmd->return_code and for my test it > > > > > currently > > > > > enters an infinite loop as a result. > > > > > > I meant to fix that. > > > > > > > > > > > > > I haven't checked other paths, but to my mind it is not a good idea > > > > > to require > > > > > two levels of error checking - the example here proves how easy it is > > > > > to forget > > > > > one. > > > > > > Demonstrably, you're correct. I think it would be good to have a kernel > > > only > > > mbox command that does the error checking though. Let me type something > > > up and > > > see how it looks. > > > > Hi Jonathan. What do you think of this? The bit I'm on the fence about is > > if I > > should validate output size too. I like the simplicity as it is, but it > > requir
Re: [PATCH v2 6/8] cxl/mem: Enable commands via CEL
On 21-02-11 12:02:15, Jonathan Cameron wrote:
> On Tue, 9 Feb 2021 16:02:57 -0800
> Ben Widawsky wrote:
>
> > CXL devices identified by the memory-device class code must implement
> > the Device Command Interface (described in 8.2.9 of the CXL 2.0 spec).
> > While the driver already maintains a list of commands it supports, there
> > is still a need to be able to distinguish between commands that the
> > driver knows about from commands that are optionally supported by the
> > hardware.
> >
> > The Command Effects Log (CEL) is specified in the CXL 2.0 specification.
> > The CEL is one of two types of logs, the other being vendor specific.
>
> I'd say "vendor specific debug" just so that no one thinks it has anything
> to do with the rest of this description (which mentioned vendor specific
> commands).
>
> > They are distinguished in hardware/spec via UUID. The CEL is useful for
> > 2 things:
> > 1. Determine which optional commands are supported by the CXL device.
> > 2. Enumerate any vendor specific commands
> >
> > The CEL is used by the driver to determine which commands are available
> > in the hardware and therefore which commands userspace is allowed to
> > execute. The set of enabled commands might be a subset of commands which
> > are advertised in UAPI via CXL_MEM_SEND_COMMAND IOCTL.
> >
> > The implementation leaves the statically defined table of commands and
> > supplements it with a bitmap to determine commands that are enabled.
> > This organization was chosen for the following reasons:
> > - Smaller memory footprint. Doesn't need a table per device.
> > - Reduce memory allocation complexity.
> > - Fixed command IDs to opcode mapping for all devices makes development
> > and debugging easier.
> > - Certain helpers are easily achievable, like cxl_for_each_cmd().
> >
> > Signed-off-by: Ben Widawsky
> > Reviewed-by: Dan Williams
> > ---
> > drivers/cxl/cxl.h| 2 +
> > drivers/cxl/mem.c| 216 +++
> > include/uapi/linux/cxl_mem.h | 1 +
> > 3 files changed, 219 insertions(+)
> >
> > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> > index b3c56fa6e126..9a5e595abfa4 100644
> > --- a/drivers/cxl/cxl.h
> > +++ b/drivers/cxl/cxl.h
> > @@ -68,6 +68,7 @@ struct cxl_memdev;
> > *(CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
> > * @mbox_mutex: Mutex to synchronize mailbox access.
> > * @firmware_version: Firmware version for the memory device.
> > + * @enabled_commands: Hardware commands found enabled in CEL.
> > * @pmem: Persistent memory capacity information.
> > * @ram: Volatile memory capacity information.
> > */
> > @@ -83,6 +84,7 @@ struct cxl_mem {
> > size_t payload_size;
> > struct mutex mbox_mutex; /* Protects device mailbox and firmware */
> > char firmware_version[0x10];
> > + unsigned long *enabled_cmds;
> >
> > struct {
> > struct range range;
> > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > index 6d766a994dce..e9aa6ca18d99 100644
> > --- a/drivers/cxl/mem.c
> > +++ b/drivers/cxl/mem.c
> > @@ -45,6 +45,8 @@ enum opcode {
> > CXL_MBOX_OP_INVALID = 0x,
> > CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
> > CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
> > + CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
> > + CXL_MBOX_OP_GET_LOG = 0x0401,
> > CXL_MBOX_OP_IDENTIFY= 0x4000,
> > CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
> > CXL_MBOX_OP_SET_LSA = 0x4103,
> > @@ -103,6 +105,19 @@ static DEFINE_IDA(cxl_memdev_ida);
> > static struct dentry *cxl_debugfs;
> > static bool raw_allow_all;
> >
> > +enum {
> > + CEL_UUID,
> > + VENDOR_DEBUG_UUID
>
> Who wants to take a bet this will get extended at somepoint in the future?
> Add a trailing comma to make that less noisy.
>
> They would never have used a UUID if this wasn't expected to expand.
> CXL spec calls out that "The following Log Identifier UUIDs are defined in
> _this_
> specification" rather implying other specs may well define more.
> Fun for the future!
>
> > +};
> > +
> > +/* See CXL 2.0 Table 170. Get Log Input Payload */
> > +static const uuid_t log_uuid[] = {
> > + [CEL_UUID] = UUID_INIT(0xda9c0b5, 0xbf41, 0x4b78, 0x8f, 0x79, 0x96,
> > + 0xb1, 0x62, 0x3b, 0x3f, 0x17),
> > + [VENDOR_DEBUG_UUID] =
Re: [PATCH v2 4/8] cxl/mem: Add basic IOCTL interface
On 21-02-11 10:06:46, Jonathan Cameron wrote:
> On Wed, 10 Feb 2021 20:40:52 -0800
> Dan Williams wrote:
>
> > On Wed, Feb 10, 2021 at 10:47 AM Jonathan Cameron
> > wrote:
> > [..]
> > > > +#define CXL_CMDS
> > > >\
> > > > + ___C(INVALID, "Invalid Command"),
> > > > \
> > > > + ___C(IDENTIFY, "Identify Command"),
> > > > \
> > > > + ___C(MAX, "Last command")
> > > > +
> > > > +#define ___C(a, b) CXL_MEM_COMMAND_ID_##a
> > > > +enum { CXL_CMDS };
> > > > +
> > > > +#undef ___C
> > > > +#define ___C(a, b) { b }
> > > > +static const struct {
> > > > + const char *name;
> > > > +} cxl_command_names[] = { CXL_CMDS };
> > > > +#undef ___C
> > >
> > > Unless there are going to be a lot of these, I'd just write them out long
> > > hand
> > > as much more readable than the macro magic.
> >
> > This macro magic isn't new to Linux it was introduced with ftrace:
> >
> > See "cpp tricks and treats": https://lwn.net/Articles/383362/
>
> Yeah. I've dealt with that one a few times. It's very cleaver and compact
> but a PITA to debug build errors related to it.
>
> >
> > >
> > > enum {
> > > CXL_MEM_COMMAND_ID_INVALID,
> > > CXL_MEM_COMMAND_ID_IDENTIFY,
> > > CXL_MEM_COMMAND_ID_MAX
> > > };
> > >
> > > static const struct {
> > > const char *name;
> > > } cxl_command_names[] = {
> > > [CXL_MEM_COMMAND_ID_INVALID] = { "Invalid Command" },
> > > [CXL_MEM_COMMAND_ID_IDENTIFY] = { "Identify Comamnd" },
> > > /* I hope you never need the Last command to exist in here as
> > > that sounds like a bug */
> > > };
> > >
> > > That's assuming I actually figured the macro fun out correctly.
> > > To my mind it's worth doing this stuff for 'lots' no so much for 3.
> >
> > The list will continue to expand, and it eliminates the "did you
> > remember to update cxl_command_names" review burden permanently.
>
> How about a compromise. Add a comment giving how the first entry expands to
> avoid people (me at least :) having to think their way through it every time?
>
> Jonathan
>
A minor tweak while here...
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index 655fbfde97fd..dac0adb879ec 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -22,7 +22,7 @@
#define CXL_CMDS \
___C(INVALID, "Invalid Command"), \
___C(IDENTIFY, "Identify Command"), \
- ___C(MAX, "Last command")
+ ___C(MAX, "invalid / last command")
#define ___C(a, b) CXL_MEM_COMMAND_ID_##a
enum { CXL_CMDS };
@@ -32,6 +32,17 @@ enum { CXL_CMDS };
static const struct {
const char *name;
} cxl_command_names[] = { CXL_CMDS };
+
+/*
+ * Here's how this actually breaks out:
+ * cxl_command_names[] = {
+ * [CXL_MEM_COMMAND_ID_INVALID] = { "Invalid Command" },
+ * [CXL_MEM_COMMAND_ID_IDENTIFY] = { "Identify Comamnd" },
+ * ...
+ * [CXL_MEM_COMMAND_ID_MAX] = { "invalid / last command" },
+ * };
+ */
+
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH v2 2/8] cxl/mem: Find device capabilities
On 21-02-11 10:01:52, Jonathan Cameron wrote:
> On Wed, 10 Feb 2021 11:54:29 -0800
> Dan Williams wrote:
>
> > > > ...
> > > >
> > > > > +static void cxl_mem_mbox_timeout(struct cxl_mem *cxlm,
> > > > > +struct mbox_cmd *mbox_cmd)
> > > > > +{
> > > > > + struct device *dev = &cxlm->pdev->dev;
> > > > > +
> > > > > + dev_dbg(dev, "Mailbox command (opcode: %#x size: %zub) timed
> > > > > out\n",
> > > > > + mbox_cmd->opcode, mbox_cmd->size_in);
> > > > > +
> > > > > + if (IS_ENABLED(CONFIG_CXL_MEM_INSECURE_DEBUG)) {
> > > >
> > > > Hmm. Whilst I can see the advantage of this for debug, I'm not sure we
> > > > want
> > > > it upstream even under a rather evil looking CONFIG variable.
> > > >
> > > > Is there a bigger lock we can use to avoid chance of accidental
> > > > enablement?
> > >
> > > Any suggestions? I'm told this functionality was extremely valuable for
> > > NVDIMM,
> > > though I haven't personally experienced it.
> >
> > Yeah, there was no problem with the identical mechanism in LIBNVDIMM
> > land. However, I notice that the useful feature for LIBNVDIMM is the
> > option to dump all payloads. This one only fires on timeouts which is
> > less useful. So I'd say fix it to dump all payloads on the argument
> > that the safety mechanism was proven with the LIBNVDIMM precedent, or
> > delete it altogether to maintain v5.12 momentum. Payload dumping can
> > be added later.
>
> I think I'd drop it for now - feels like a topic that needs more discussion.
>
> Also, dumping this data to the kernel log isn't exactly elegant - particularly
> if we dump a lot more of it. Perhaps tracepoints?
>
I'll drop it. It's also a small enough bit to add on for developers. When I post
v3, I will add that bit on top as an RFC. My personal preference FWIW is to use
debugfs to store the payload of the last executed command.
We went with this because of the mechanism's provenance (libnvdimm)
> >
> > [..]
> > > > > diff --git a/include/uapi/linux/pci_regs.h
> > > > > b/include/uapi/linux/pci_regs.h
> > > > > index e709ae8235e7..6267ca9ae683 100644
> > > > > --- a/include/uapi/linux/pci_regs.h
> > > > > +++ b/include/uapi/linux/pci_regs.h
> > > > > @@ -1080,6 +1080,7 @@
> > > > >
> > > > > /* Designated Vendor-Specific (DVSEC, PCI_EXT_CAP_ID_DVSEC) */
> > > > > #define PCI_DVSEC_HEADER1 0x4 /* Designated Vendor-Specific
> > > > > Header1 */
> > > > > +#define PCI_DVSEC_HEADER1_LENGTH_MASK 0xFFF0
> > > >
> > > > Seems sensible to add the revision mask as well.
> > > > The vendor id currently read using a word read rather than dword, but
> > > > perhaps
> > > > neater to add that as well for completeness?
> > > >
> > > > Having said that, given Bjorn's comment on clashes and the fact he'd
> > > > rather see
> > > > this stuff defined in drivers and combined later (see review patch 1
> > > > and follow
> > > > the link) perhaps this series should not touch this header at all.
> > >
> > > I'm fine to move it back.
> >
> > Yeah, we're playing tennis now between Bjorn's and Christoph's
> > comments, but I like Bjorn's suggestion of "deduplicate post merge"
> > given the bloom of DVSEC infrastructure landing at the same time.
> I guess it may depend on timing of this. Personally I think 5.12 may be too
> aggressive.
>
> As long as Bjorn can take a DVSEC deduplication as an immutable branch then
> perhaps
> during 5.13 this tree can sit on top of that.
>
> Jonathan
>
>
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH v2 5/8] cxl/mem: Add a "RAW" send command
On 21-02-11 11:19:24, Jonathan Cameron wrote: > On Tue, 9 Feb 2021 16:02:56 -0800 > Ben Widawsky wrote: > > > The CXL memory device send interface will have a number of supported > > commands. The raw command is not such a command. Raw commands allow > > userspace to send a specified opcode to the underlying hardware and > > bypass all driver checks on the command. This is useful for a couple of > > usecases, mainly: > > 1. Undocumented vendor specific hardware commands > > This one I get. There are things we'd love to standardize but often they > need proving in a generation of hardware before the data is available to > justify taking it to a standards body. Stuff like performance stats. > This stuff will all sit in the vendor defined range. Maybe there is an > argument for in driver hooks to allow proper support even for these > (Ben mentioned this in the other branch of the thread). > > > 2. Prototyping new hardware commands not yet supported by the driver > > For 2, could just have a convenient place to enable this by one line patch. > Some subsystems (SPI comes to mind) do this for their equivalent of raw > commands. The code is all there to enable it but you need to hook it > up if you want to use it. Avoids chance of a distro shipping it. > I'm fine to drop #2 as a justification point, or maybe reword the commit message to say, "you could also just do... but since we have it for #1 already..." > > > > While this all sounds very powerful it comes with a couple of caveats: > > 1. Bug reports using raw commands will not get the same level of > >attention as bug reports using supported commands (via taint). > > 2. Supported commands will be rejected by the RAW command. > > Perhaps I'm missing reading this point 2 (not sure the code actually does it!) > > As stated what worries me as it means when we add support for a new > bit of the spec we just broke the userspace ABI. > It does not break ABI. The agreement is userspace must always use the QUERY command to find out what commands are supported. If it tries to use a RAW command that is a supported command, it will be rejected. In the case you mention, that's an application bug. If there is a way to document that better than what's already in the UAPI kdocs, I'm open to suggestions. Unlike perhaps other UAPI, this one only promises to give you a way to determine what commands you can use, not the list of what commands you can use. > > > > With this comes new debugfs knob to allow full access to your toes with > > your weapon of choice. > > A few trivial things inline, > > Jonathan > > > > > Cc: Ariel Sibley > > Signed-off-by: Ben Widawsky > > Reviewed-by: Dan Williams > > --- > > drivers/cxl/Kconfig | 18 + > > drivers/cxl/mem.c| 125 ++- > > include/uapi/linux/cxl_mem.h | 12 +++- > > 3 files changed, 152 insertions(+), 3 deletions(-) > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > index c4ba3aa0a05d..08eaa8e52083 100644 > > --- a/drivers/cxl/Kconfig > > +++ b/drivers/cxl/Kconfig > > @@ -33,6 +33,24 @@ config CXL_MEM > > > > If unsure say 'm'. > > > > +config CXL_MEM_RAW_COMMANDS > > + bool "RAW Command Interface for Memory Devices" > > + depends on CXL_MEM > > + help > > + Enable CXL RAW command interface. > > + > > + The CXL driver ioctl interface may assign a kernel ioctl command > > + number for each specification defined opcode. At any given point in > > + time the number of opcodes that the specification defines and a device > > + may implement may exceed the kernel's set of associated ioctl function > > + numbers. The mismatch is either by omission, specification is too new, > > + or by design. When prototyping new hardware, or developing / debugging > > + the driver it is useful to be able to submit any possible command to > > + the hardware, even commands that may crash the kernel due to their > > + potential impact to memory currently in use by the kernel. > > + > > + If developing CXL hardware or the driver say Y, otherwise say N. > > + > > config CXL_MEM_INSECURE_DEBUG > > bool "CXL.mem debugging" > > depends on CXL_MEM > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c > > index ce65630bb75e..6d766a994dce 100644 > > --- a/drivers/cxl/mem.c > > +++ b/drivers/cxl/mem.c > > @@ -1,6 +1,8 @@ > > // SPDX-License-Identifier: GPL-2.0-only
Re: [PATCH v2 2/8] cxl/mem: Find device capabilities
On 21-02-11 09:55:48, Jonathan Cameron wrote: > On Wed, 10 Feb 2021 10:16:05 -0800 > Ben Widawsky wrote: > > > On 21-02-10 08:55:57, Ben Widawsky wrote: > > > On 21-02-10 15:07:59, Jonathan Cameron wrote: > > > > On Wed, 10 Feb 2021 13:32:52 + > > > > Jonathan Cameron wrote: > > > > > > > > > On Tue, 9 Feb 2021 16:02:53 -0800 > > > > > Ben Widawsky wrote: > > > > > > > > > > > Provide enough functionality to utilize the mailbox of a memory > > > > > > device. > > > > > > The mailbox is used to interact with the firmware running on the > > > > > > memory > > > > > > device. The flow is proven with one implemented command, "identify". > > > > > > Because the class code has already told the driver this is a memory > > > > > > device and the identify command is mandatory. > > > > > > > > > > > > CXL devices contain an array of capabilities that describe the > > > > > > interactions software can have with the device or firmware running > > > > > > on > > > > > > the device. A CXL compliant device must implement the device status > > > > > > and > > > > > > the mailbox capability. Additionally, a CXL compliant memory device > > > > > > must > > > > > > implement the memory device capability. Each of the capabilities can > > > > > > [will] provide an offset within the MMIO region for interacting > > > > > > with the > > > > > > CXL device. > > > > > > > > > > > > The capabilities tell the driver how to find and map the register > > > > > > space > > > > > > for CXL Memory Devices. The registers are required to utilize the > > > > > > CXL > > > > > > spec defined mailbox interface. The spec outlines two mailboxes, > > > > > > primary > > > > > > and secondary. The secondary mailbox is earmarked for system > > > > > > firmware, > > > > > > and not handled in this driver. > > > > > > > > > > > > Primary mailboxes are capable of generating an interrupt when > > > > > > submitting > > > > > > a background command. That implementation is saved for a later time. > > > > > > > > > > > > Link: https://www.computeexpresslink.org/download-the-specification > > > > > > Signed-off-by: Ben Widawsky > > > > > > Reviewed-by: Dan Williams > > > > > > > > > > Hi Ben, > > > > > > > > > > > > > > > > +/** > > > > > > + * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory > > > > > > device. > > > > > > + * @cxlm: The CXL memory device to communicate with. > > > > > > + * @mbox_cmd: Command to send to the memory device. > > > > > > + * > > > > > > + * Context: Any context. Expects mbox_lock to be held. > > > > > > + * Return: -ETIMEDOUT if timeout occurred waiting for completion. > > > > > > 0 on success. > > > > > > + * Caller should check the return code in @mbox_cmd to > > > > > > make sure it > > > > > > + * succeeded. > > > > > > > > > > cxl_xfer_log() doesn't check mbox_cmd->return_code and for my test it > > > > > currently > > > > > enters an infinite loop as a result. > > > > > > I meant to fix that. > > > > > > > > > > > > > I haven't checked other paths, but to my mind it is not a good idea > > > > > to require > > > > > two levels of error checking - the example here proves how easy it is > > > > > to forget > > > > > one. > > > > > > Demonstrably, you're correct. I think it would be good to have a kernel > > > only > > > mbox command that does the error checking though. Let me type something > > > up and > > > see how it looks. > > > > Hi Jonathan. What do you think of this? The bit I'm on the fence about is > > if I > > should validate output size too. I like the simplicity as it is, but it > > requires &
Re: [PATCH v2 4/8] cxl/mem: Add basic IOCTL interface
On 21-02-10 18:45:40, Jonathan Cameron wrote:
> On Tue, 9 Feb 2021 16:02:55 -0800
> Ben Widawsky wrote:
>
> > Add a straightforward IOCTL that provides a mechanism for userspace to
> > query the supported memory device commands. CXL commands as they appear
> > to userspace are described as part of the UAPI kerneldoc. The command
> > list returned via this IOCTL will contain the full set of commands that
> > the driver supports, however, some of those commands may not be
> > available for use by userspace.
> >
> > Memory device commands first appear in the CXL 2.0 specification. They
> > are submitted through a mailbox mechanism specified also originally
> > specified in the CXL 2.0 specification.
> >
> > The send command allows userspace to issue mailbox commands directly to
> > the hardware. The list of available commands to send are the output of
> > the query command. The driver verifies basic properties of the command
> > and possibly inspect the input (or output) payload to determine whether
> > or not the command is allowed (or might taint the kernel).
> >
> > Reported-by: kernel test robot # bug in earlier revision
> > Signed-off-by: Ben Widawsky
> > Reviewed-by: Dan Williams
>
> A bit of anti macro commentary below. Heavy use of them may make the code
> shorter, but I'd argue they make it harder to do review if you've not looked
> at a given bit of code for a while.
>
> Also there is a bit of documentation in here for flags that don't seem to
> exist (at this stage anyway) - may just be in the wrong patch.
>
> Jonathan
>
>
> > ---
> > .clang-format | 1 +
> > .../userspace-api/ioctl/ioctl-number.rst | 1 +
> > drivers/cxl/mem.c | 291 +-
> > include/uapi/linux/cxl_mem.h | 152 +
> > 4 files changed, 443 insertions(+), 2 deletions(-)
> > create mode 100644 include/uapi/linux/cxl_mem.h
> >
> > diff --git a/.clang-format b/.clang-format
> > index 10dc5a9a61b3..3f11c8901b43 100644
> > --- a/.clang-format
> > +++ b/.clang-format
> > @@ -109,6 +109,7 @@ ForEachMacros:
> >- 'css_for_each_child'
> >- 'css_for_each_descendant_post'
> >- 'css_for_each_descendant_pre'
> > + - 'cxl_for_each_cmd'
> >- 'device_for_each_child_node'
> >- 'dma_fence_chain_for_each'
> >- 'do_for_each_ftrace_op'
> > diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst
> > b/Documentation/userspace-api/ioctl/ioctl-number.rst
> > index a4c75a28c839..6eb8e634664d 100644
> > --- a/Documentation/userspace-api/ioctl/ioctl-number.rst
> > +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
> > @@ -352,6 +352,7 @@ Code Seq#Include File
> > Comments
> >
> > <mailto:michael.kl...@puffin.lb.shuttle.de>
> > 0xCC 00-0F drivers/misc/ibmvmc.h
> > pseries VMC driver
> > 0xCD 01 linux/reiserfs_fs.h
> > +0xCE 01-02 uapi/linux/cxl_mem.h
> > Compute Express Link Memory Devices
> > 0xCF 02 fs/cifs/ioctl.c
> > 0xDB 00-0F drivers/char/mwave/mwavepub.h
> > 0xDD 00-3F ZFCP
> > device driver see drivers/s390/scsi/
> > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > index 8bbd2495e237..ce65630bb75e 100644
> > --- a/drivers/cxl/mem.c
> > +++ b/drivers/cxl/mem.c
> > @@ -1,5 +1,6 @@
> > // SPDX-License-Identifier: GPL-2.0-only
> > /* Copyright(c) 2020 Intel Corporation. All rights reserved. */
> > +#include
> > #include
> > #include
> > #include
> > @@ -39,6 +40,7 @@
> > #define CXL_MAILBOX_TIMEOUT_MS (2 * HZ)
> >
> > enum opcode {
> > + CXL_MBOX_OP_INVALID = 0x,
> > CXL_MBOX_OP_IDENTIFY= 0x4000,
> > CXL_MBOX_OP_MAX = 0x1
> > };
> > @@ -90,9 +92,57 @@ struct cxl_memdev {
> > static int cxl_mem_major;
> > static DEFINE_IDA(cxl_memdev_ida);
> >
> > +/**
> > + * struct cxl_mem_command - Driver representation of a memory device
> > command
> > + * @info: Command information as it exists for the UAPI
> > + * @opcode: The actual bits used for the mailbox protocol
> > + * @flags: Set of flags reflecting the state o
Re: [PATCH v2 2/8] cxl/mem: Find device capabilities
On 21-02-10 13:32:52, Jonathan Cameron wrote:
> On Tue, 9 Feb 2021 16:02:53 -0800
> Ben Widawsky wrote:
>
> > Provide enough functionality to utilize the mailbox of a memory device.
> > The mailbox is used to interact with the firmware running on the memory
> > device. The flow is proven with one implemented command, "identify".
> > Because the class code has already told the driver this is a memory
> > device and the identify command is mandatory.
> >
> > CXL devices contain an array of capabilities that describe the
> > interactions software can have with the device or firmware running on
> > the device. A CXL compliant device must implement the device status and
> > the mailbox capability. Additionally, a CXL compliant memory device must
> > implement the memory device capability. Each of the capabilities can
> > [will] provide an offset within the MMIO region for interacting with the
> > CXL device.
> >
> > The capabilities tell the driver how to find and map the register space
> > for CXL Memory Devices. The registers are required to utilize the CXL
> > spec defined mailbox interface. The spec outlines two mailboxes, primary
> > and secondary. The secondary mailbox is earmarked for system firmware,
> > and not handled in this driver.
> >
> > Primary mailboxes are capable of generating an interrupt when submitting
> > a background command. That implementation is saved for a later time.
> >
> > Link: https://www.computeexpresslink.org/download-the-specification
> > Signed-off-by: Ben Widawsky
> > Reviewed-by: Dan Williams
>
> Hi Ben,
>
>
> > +/**
> > + * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory device.
> > + * @cxlm: The CXL memory device to communicate with.
> > + * @mbox_cmd: Command to send to the memory device.
> > + *
> > + * Context: Any context. Expects mbox_lock to be held.
> > + * Return: -ETIMEDOUT if timeout occurred waiting for completion. 0 on
> > success.
> > + * Caller should check the return code in @mbox_cmd to make sure it
> > + * succeeded.
>
> cxl_xfer_log() doesn't check mbox_cmd->return_code and for my test it
> currently
> enters an infinite loop as a result.
>
> I haven't checked other paths, but to my mind it is not a good idea to require
> two levels of error checking - the example here proves how easy it is to
> forget
> one.
>
> Now all I have to do is figure out why I'm getting an error in the first
> place!
>
> Jonathan
>
>
>
> > + *
> > + * This is a generic form of the CXL mailbox send command, thus the only
> > I/O
> > + * operations used are cxl_read_mbox_reg(). Memory devices, and perhaps
> > other
> > + * types of CXL devices may have further information available upon error
> > + * conditions.
> > + *
> > + * The CXL spec allows for up to two mailboxes. The intention is for the
> > primary
> > + * mailbox to be OS controlled and the secondary mailbox to be used by
> > system
> > + * firmware. This allows the OS and firmware to communicate with the
> > device and
> > + * not need to coordinate with each other. The driver only uses the primary
> > + * mailbox.
> > + */
> > +static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
> > +struct mbox_cmd *mbox_cmd)
> > +{
> > + void __iomem *payload = cxlm->mbox_regs + CXLDEV_MBOX_PAYLOAD_OFFSET;
> > + u64 cmd_reg, status_reg;
> > + size_t out_len;
> > + int rc;
> > +
> > + lockdep_assert_held(&cxlm->mbox_mutex);
> > +
> > + /*
> > +* Here are the steps from 8.2.8.4 of the CXL 2.0 spec.
> > +* 1. Caller reads MB Control Register to verify doorbell is clear
> > +* 2. Caller writes Command Register
> > +* 3. Caller writes Command Payload Registers if input payload is
> > non-empty
> > +* 4. Caller writes MB Control Register to set doorbell
> > +* 5. Caller either polls for doorbell to be clear or waits for
> > interrupt if configured
> > +* 6. Caller reads MB Status Register to fetch Return code
> > +* 7. If command successful, Caller reads Command Register to get
> > Payload Length
> > +* 8. If output payload is non-empty, host reads Command Payload
> > Registers
> > +*
> > +* Hardware is free to do whatever it wants before the doorbell is rung,
> > +* and isn't allowed to change anything after it clears the doorbell. As
> > +* such, steps 2 and 3 can happen
Re: [PATCH v2 5/8] cxl/mem: Add a "RAW" send command
On 21-02-10 18:46:04, ariel.sib...@microchip.com wrote: > > > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > > > > > index c4ba3aa0a05d..08eaa8e52083 100644 > > > > > > --- a/drivers/cxl/Kconfig > > > > > > +++ b/drivers/cxl/Kconfig > > > > > > @@ -33,6 +33,24 @@ config CXL_MEM > > > > > > > > > > > > If unsure say 'm'. > > > > > > > > > > > > +config CXL_MEM_RAW_COMMANDS > > > > > > + bool "RAW Command Interface for Memory Devices" > > > > > > + depends on CXL_MEM > > > > > > + help > > > > > > + Enable CXL RAW command interface. > > > > > > + > > > > > > + The CXL driver ioctl interface may assign a kernel ioctl > > > > > > command > > > > > > + number for each specification defined opcode. At any > > > > > > given point in > > > > > > + time the number of opcodes that the specification defines > > > > > > and a device > > > > > > + may implement may exceed the kernel's set of associated > > > > > > ioctl function > > > > > > + numbers. The mismatch is either by omission, > > > > > > specification is too new, > > > > > > + or by design. When prototyping new hardware, or > > > > > > developing / > > > > > > debugging > > > > > > + the driver it is useful to be able to submit any possible > > > > > > command to > > > > > > + the hardware, even commands that may crash the kernel due > > > > > > to their > > > > > > + potential impact to memory currently in use by the kernel. > > > > > > + > > > > > > + If developing CXL hardware or the driver say Y, otherwise > > > > > > say N. > > > > > > > > > > Blocking RAW commands by default will prevent vendors from developing > > > > > user > > > > > space tools that utilize vendor specific commands. Vendors of CXL.mem > > > > > devices > > > > > should take ownership of ensuring any vendor defined commands that > > > > > could cause > > > > > user data to be exposed or corrupted are disabled at the device level > > > > > for > > > > > shipping configurations. > > > > > > > > Thanks for brining this up Ariel. If there is a recommendation on how > > > > to codify > > > > this, I would certainly like to know because the explanation will be > > > > long. > > > > > > > > --- > > > > > > > > The background: > > > > > > > > The enabling/disabling of the Kconfig option is driven by the > > > > distribution > > > > and/or system integrator. Even if we made the default 'y', nothing > > > > stops them > > > > from changing that. if you are using this driver in production and > > > > insist on > > > > using RAW commands, you are free to carry around a small patch to get > > > > rid of the > > > > WARN (it is a one-liner). > > > > > > > > To recap why this is in place - the driver owns the sanctity of the > > > > device and > > > > therefore a [large] part of the whole system. What we can do as driver > > > > writers > > > > is figure out the set of commands that are "safe" and allow those. > > > > Aside from > > > > being able to validate them, we're able to mediate them with other > > > > parallel > > > > operations that might conflict. We gain the ability to squint extra > > > > hard at bug > > > > reports. We provide a reason to try to use a well defined part of the > > > > spec. > > > > Realizing that only allowing that small set of commands in a rapidly > > > > growing > > > > ecosystem is not a welcoming API; we decided on RAW. > > > > > > > > Vendor commands can be one of two types: > > > > 1. Some functionality probably most vendors want. > > > > 2. Functionality that is really single vendor specific. > > > > > > > > Hopefully we can agree that the path for case #1 is to work with the > > > > consortium > > > > to standardize a command that does what is needed and that can > > > > eventually become > > > > part of UAPI. The situation is unfortunate, but temporary. If you won't > > > > be able > > > > to upgrade your kernel, patch out the WARN as above. > > > > > > > > The second situation is interesting and does need some more thought and > > > > discussion. > > > > > > > > --- > > > > > > > > I see 3 realistic options for truly vendor specific commands. > > > > 1. Tough noogies. Vendors aren't special and they shouldn't do that. > > > > 2. modparam to disable the WARN for specific devices (let the sysadmin > > > > decide) > > > > 3. Try to make them part of UAPI. > > > > > > > > The right answer to me is #1, but I also realize I live in the real > > > > world. > > > > > > > > #2 provides too much flexibility. Vendors will just do what they please > > > > and > > > > distros and/or integrators will be seen as hostile if they don't > > > > accommodate. > > > > > > > > I like #3, but I have a feeling not everyone will agree. My proposal > > > > for vendor > > > > specific commands is, if it's clear it's truly a unique command, allow > > > > adding it > > > > as part of UAPI (moving it out of RAW). I expect
Re: [PATCH v2 2/8] cxl/mem: Find device capabilities
On 21-02-10 17:41:04, Jonathan Cameron wrote: > On Tue, 9 Feb 2021 16:02:53 -0800 > Ben Widawsky wrote: > > > Provide enough functionality to utilize the mailbox of a memory device. > > The mailbox is used to interact with the firmware running on the memory > > device. The flow is proven with one implemented command, "identify". > > Because the class code has already told the driver this is a memory > > device and the identify command is mandatory. > > > > CXL devices contain an array of capabilities that describe the > > interactions software can have with the device or firmware running on > > the device. A CXL compliant device must implement the device status and > > the mailbox capability. Additionally, a CXL compliant memory device must > > implement the memory device capability. Each of the capabilities can > > [will] provide an offset within the MMIO region for interacting with the > > CXL device. > > > > The capabilities tell the driver how to find and map the register space > > for CXL Memory Devices. The registers are required to utilize the CXL > > spec defined mailbox interface. The spec outlines two mailboxes, primary > > and secondary. The secondary mailbox is earmarked for system firmware, > > and not handled in this driver. > > > > Primary mailboxes are capable of generating an interrupt when submitting > > a background command. That implementation is saved for a later time. > > > > Link: https://www.computeexpresslink.org/download-the-specification > > Signed-off-by: Ben Widawsky > > Reviewed-by: Dan Williams > > A few more comments inline (proper review whereas my other reply was a > bug chase). > > Jonathan > > > --- > > drivers/cxl/Kconfig | 14 + > > drivers/cxl/cxl.h | 93 +++ > > drivers/cxl/mem.c | 511 +- > > drivers/cxl/pci.h | 13 + > > include/uapi/linux/pci_regs.h | 1 + > > 5 files changed, 630 insertions(+), 2 deletions(-) > > create mode 100644 drivers/cxl/cxl.h > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > index 9e80b311e928..c4ba3aa0a05d 100644 > > --- a/drivers/cxl/Kconfig > > +++ b/drivers/cxl/Kconfig > > @@ -32,4 +32,18 @@ config CXL_MEM > > Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification. > > > > If unsure say 'm'. > > + > > +config CXL_MEM_INSECURE_DEBUG > > + bool "CXL.mem debugging" > > As mentioned below, this makes me a tiny bit uncomfortable. > > > + depends on CXL_MEM > > + help > > + Enable debug of all CXL command payloads. > > + > > + Some CXL devices and controllers support encryption and other > > + security features. The payloads for the commands that enable > > + those features may contain sensitive clear-text security > > + material. Disable debug of those command payloads by default. > > + If you are a kernel developer actively working on CXL > > + security enabling say Y, otherwise say N. > > + > > endif > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > > new file mode 100644 > > index ..745f5e0bfce3 > > --- /dev/null > > +++ b/drivers/cxl/cxl.h > > @@ -0,0 +1,93 @@ > > +/* SPDX-License-Identifier: GPL-2.0-only */ > > +/* Copyright(c) 2020 Intel Corporation. */ > > + > > +#ifndef __CXL_H__ > > +#define __CXL_H__ > > + > > +#include > > +#include > > +#include > > + > > +/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */ > > +#define CXLDEV_CAP_ARRAY_OFFSET 0x0 > > +#define CXLDEV_CAP_ARRAY_CAP_ID 0 > > +#define CXLDEV_CAP_ARRAY_ID_MASK GENMASK(15, 0) > > +#define CXLDEV_CAP_ARRAY_COUNT_MASK GENMASK(47, 32) > > +/* CXL 2.0 8.2.8.2.1 CXL Device Capabilities */ > > +#define CXLDEV_CAP_CAP_ID_DEVICE_STATUS 0x1 > > +#define CXLDEV_CAP_CAP_ID_PRIMARY_MAILBOX 0x2 > > +#define CXLDEV_CAP_CAP_ID_SECONDARY_MAILBOX 0x3 > > +#define CXLDEV_CAP_CAP_ID_MEMDEV 0x4000 > > + > > +/* CXL 2.0 8.2.8.4 Mailbox Registers */ > > +#define CXLDEV_MBOX_CAPS_OFFSET 0x00 > > +#define CXLDEV_MBOX_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0) > > +#define CXLDEV_MBOX_CTRL_OFFSET 0x04 > > +#define CXLDEV_MBOX_CTRL_DOORBELL BIT(0) > > +#define CXLDEV_MBOX_CMD_OFFSET 0x08 > > +#define CXLDEV_MBOX_CMD_COMMAND_OPCODE_MASK GENMASK(15, 0) > > +#define CXLDEV_MBOX_CMD_PAYLOAD_LENGTH_MASK GENMASK(36, 16) > >
Re: [PATCH v2 2/8] cxl/mem: Find device capabilities
On 21-02-10 08:55:57, Ben Widawsky wrote: > On 21-02-10 15:07:59, Jonathan Cameron wrote: > > On Wed, 10 Feb 2021 13:32:52 + > > Jonathan Cameron wrote: > > > > > On Tue, 9 Feb 2021 16:02:53 -0800 > > > Ben Widawsky wrote: > > > > > > > Provide enough functionality to utilize the mailbox of a memory device. > > > > The mailbox is used to interact with the firmware running on the memory > > > > device. The flow is proven with one implemented command, "identify". > > > > Because the class code has already told the driver this is a memory > > > > device and the identify command is mandatory. > > > > > > > > CXL devices contain an array of capabilities that describe the > > > > interactions software can have with the device or firmware running on > > > > the device. A CXL compliant device must implement the device status and > > > > the mailbox capability. Additionally, a CXL compliant memory device must > > > > implement the memory device capability. Each of the capabilities can > > > > [will] provide an offset within the MMIO region for interacting with the > > > > CXL device. > > > > > > > > The capabilities tell the driver how to find and map the register space > > > > for CXL Memory Devices. The registers are required to utilize the CXL > > > > spec defined mailbox interface. The spec outlines two mailboxes, primary > > > > and secondary. The secondary mailbox is earmarked for system firmware, > > > > and not handled in this driver. > > > > > > > > Primary mailboxes are capable of generating an interrupt when submitting > > > > a background command. That implementation is saved for a later time. > > > > > > > > Link: https://www.computeexpresslink.org/download-the-specification > > > > Signed-off-by: Ben Widawsky > > > > Reviewed-by: Dan Williams > > > > > > Hi Ben, > > > > > > > > > > +/** > > > > + * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory device. > > > > + * @cxlm: The CXL memory device to communicate with. > > > > + * @mbox_cmd: Command to send to the memory device. > > > > + * > > > > + * Context: Any context. Expects mbox_lock to be held. > > > > + * Return: -ETIMEDOUT if timeout occurred waiting for completion. 0 on > > > > success. > > > > + * Caller should check the return code in @mbox_cmd to make > > > > sure it > > > > + * succeeded. > > > > > > cxl_xfer_log() doesn't check mbox_cmd->return_code and for my test it > > > currently > > > enters an infinite loop as a result. > > I meant to fix that. > > > > > > > I haven't checked other paths, but to my mind it is not a good idea to > > > require > > > two levels of error checking - the example here proves how easy it is to > > > forget > > > one. > > Demonstrably, you're correct. I think it would be good to have a kernel only > mbox command that does the error checking though. Let me type something up and > see how it looks. Hi Jonathan. What do you think of this? The bit I'm on the fence about is if I should validate output size too. I like the simplicity as it is, but it requires every caller to possibly check output size, which is kind of the same problem you're originally pointing out. diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c index 55c5f5a6023f..ad7b2077ab28 100644 --- a/drivers/cxl/mem.c +++ b/drivers/cxl/mem.c @@ -284,7 +284,7 @@ static void cxl_mem_mbox_timeout(struct cxl_mem *cxlm, } /** - * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory device. + * __cxl_mem_mbox_send_cmd() - Execute a mailbox command * @cxlm: The CXL memory device to communicate with. * @mbox_cmd: Command to send to the memory device. * @@ -296,7 +296,8 @@ static void cxl_mem_mbox_timeout(struct cxl_mem *cxlm, * This is a generic form of the CXL mailbox send command, thus the only I/O * operations used are cxl_read_mbox_reg(). Memory devices, and perhaps other * types of CXL devices may have further information available upon error - * conditions. + * conditions. Driver facilities wishing to send mailbox commands should use the + * wrapper command. * * The CXL spec allows for up to two mailboxes. The intention is for the primary * mailbox to be OS controlled and the secondary mailbox to be used by system @@ -304,8 +305,8 @@ static void cxl_mem_mbox_timeout(struct cxl_mem *cxlm,
Re: [PATCH v2 5/8] cxl/mem: Add a "RAW" send command
On 21-02-10 18:03:35, ariel.sib...@microchip.com wrote: > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > > > index c4ba3aa0a05d..08eaa8e52083 100644 > > > > --- a/drivers/cxl/Kconfig > > > > +++ b/drivers/cxl/Kconfig > > > > @@ -33,6 +33,24 @@ config CXL_MEM > > > > > > > > If unsure say 'm'. > > > > > > > > +config CXL_MEM_RAW_COMMANDS > > > > + bool "RAW Command Interface for Memory Devices" > > > > + depends on CXL_MEM > > > > + help > > > > + Enable CXL RAW command interface. > > > > + > > > > + The CXL driver ioctl interface may assign a kernel ioctl > > > > command > > > > + number for each specification defined opcode. At any given > > > > point in > > > > + time the number of opcodes that the specification defines and > > > > a device > > > > + may implement may exceed the kernel's set of associated ioctl > > > > function > > > > + numbers. The mismatch is either by omission, specification is > > > > too new, > > > > + or by design. When prototyping new hardware, or developing / > > > > debugging > > > > + the driver it is useful to be able to submit any possible > > > > command to > > > > + the hardware, even commands that may crash the kernel due to > > > > their > > > > + potential impact to memory currently in use by the kernel. > > > > + > > > > + If developing CXL hardware or the driver say Y, otherwise say > > > > N. > > > > > > Blocking RAW commands by default will prevent vendors from developing user > > > space tools that utilize vendor specific commands. Vendors of CXL.mem > > > devices > > > should take ownership of ensuring any vendor defined commands that could > > > cause > > > user data to be exposed or corrupted are disabled at the device level for > > > shipping configurations. > > > > Thanks for brining this up Ariel. If there is a recommendation on how to > > codify > > this, I would certainly like to know because the explanation will be long. > > > > --- > > > > The background: > > > > The enabling/disabling of the Kconfig option is driven by the distribution > > and/or system integrator. Even if we made the default 'y', nothing stops > > them > > from changing that. if you are using this driver in production and insist on > > using RAW commands, you are free to carry around a small patch to get rid > > of the > > WARN (it is a one-liner). > > > > To recap why this is in place - the driver owns the sanctity of the device > > and > > therefore a [large] part of the whole system. What we can do as driver > > writers > > is figure out the set of commands that are "safe" and allow those. Aside > > from > > being able to validate them, we're able to mediate them with other parallel > > operations that might conflict. We gain the ability to squint extra hard at > > bug > > reports. We provide a reason to try to use a well defined part of the spec. > > Realizing that only allowing that small set of commands in a rapidly growing > > ecosystem is not a welcoming API; we decided on RAW. > > > > Vendor commands can be one of two types: > > 1. Some functionality probably most vendors want. > > 2. Functionality that is really single vendor specific. > > > > Hopefully we can agree that the path for case #1 is to work with the > > consortium > > to standardize a command that does what is needed and that can eventually > > become > > part of UAPI. The situation is unfortunate, but temporary. If you won't be > > able > > to upgrade your kernel, patch out the WARN as above. > > > > The second situation is interesting and does need some more thought and > > discussion. > > > > --- > > > > I see 3 realistic options for truly vendor specific commands. > > 1. Tough noogies. Vendors aren't special and they shouldn't do that. > > 2. modparam to disable the WARN for specific devices (let the sysadmin > > decide) > > 3. Try to make them part of UAPI. > > > > The right answer to me is #1, but I also realize I live in the real world. > > > > #2 provides too much flexibility. Vendors will just do what they please and > > distros and/or integrators will be seen as hostile if they don't > > accommodate. > > > > I like #3, but I have a feeling not everyone will agree. My proposal for > > vendor > > specific commands is, if it's clear it's truly a unique command, allow > > adding it > > as part of UAPI (moving it out of RAW). I expect like 5 of these, ever. If > > we > > start getting multiple per vendor, we've failed. The infrastructure is > > already > > in place to allow doing this pretty easily. I think we'd have to draw up > > some > > guidelines (like adding test cases for the command) to allow these to come > > in. > > Anything with command effects is going to need extra scrutiny. > > This would necessitate adding specific opcode values in the range C000h-h > to UAPI, and those would then be allowed for all CXL.mem devices, co
Re: [PATCH v2 1/8] cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints
On 21-02-10 16:17:07, Jonathan Cameron wrote: > On Tue, 9 Feb 2021 16:02:52 -0800 > Ben Widawsky wrote: > > > From: Dan Williams > > > > The CXL.mem protocol allows a device to act as a provider of "System > > RAM" and/or "Persistent Memory" that is fully coherent as if the memory > > was attached to the typical CPU memory controller. > > > > With the CXL-2.0 specification a PCI endpoint can implement a "Type-3" > > device interface and give the operating system control over "Host > > Managed Device Memory". See section 2.3 Type 3 CXL Device. > > > > The memory range exported by the device may optionally be described by > > the platform firmware memory map, or by infrastructure like LIBNVDIMM to > > provision persistent memory capacity from one, or more, CXL.mem devices. > > > > A pre-requisite for Linux-managed memory-capacity provisioning is this > > cxl_mem driver that can speak the mailbox protocol defined in section > > 8.2.8.4 Mailbox Registers. > > > > For now just land the initial driver boiler-plate and Documentation/ > > infrastructure. > > > > Link: https://www.computeexpresslink.org/download-the-specification > > Cc: Jonathan Corbet > > Signed-off-by: Dan Williams > > Signed-off-by: Ben Widawsky > > Acked-by: David Rientjes (v1) > > A few trivial bits inline but nothing that I feel that strongly about. > It is probably a good idea to add a note about generic dvsec code > somewhere in this patch description (to avoid people raising it on > future versions!) > > With the define of PCI_EXT_CAP_ID_DVSEC dropped (it's in the generic > header already). > > Reviewed-by: Jonathan Cameron > > > --- > > Documentation/driver-api/cxl/index.rst| 12 > > .../driver-api/cxl/memory-devices.rst | 29 + > > Documentation/driver-api/index.rst| 1 + > > drivers/Kconfig | 1 + > > drivers/Makefile | 1 + > > drivers/cxl/Kconfig | 35 +++ > > drivers/cxl/Makefile | 4 ++ > > drivers/cxl/mem.c | 63 +++ > > drivers/cxl/pci.h | 18 ++ > > include/linux/pci_ids.h | 1 + > > 10 files changed, 165 insertions(+) > > create mode 100644 Documentation/driver-api/cxl/index.rst > > create mode 100644 Documentation/driver-api/cxl/memory-devices.rst > > create mode 100644 drivers/cxl/Kconfig > > create mode 100644 drivers/cxl/Makefile > > create mode 100644 drivers/cxl/mem.c > > create mode 100644 drivers/cxl/pci.h > > > > diff --git a/Documentation/driver-api/cxl/index.rst > > b/Documentation/driver-api/cxl/index.rst > > new file mode 100644 > > index ..036e49553542 > > --- /dev/null > > +++ b/Documentation/driver-api/cxl/index.rst > > @@ -0,0 +1,12 @@ > > +.. SPDX-License-Identifier: GPL-2.0 > > + > > + > > +Compute Express Link > > + > > + > > +.. toctree:: > > + :maxdepth: 1 > > + > > + memory-devices > > + > > +.. only:: subproject and html > > diff --git a/Documentation/driver-api/cxl/memory-devices.rst > > b/Documentation/driver-api/cxl/memory-devices.rst > > new file mode 100644 > > index ..43177e700d62 > > --- /dev/null > > +++ b/Documentation/driver-api/cxl/memory-devices.rst > > @@ -0,0 +1,29 @@ > > +.. SPDX-License-Identifier: GPL-2.0 > > +.. include:: > > + > > +=== > > +Compute Express Link Memory Devices > > +=== > > + > > +A Compute Express Link Memory Device is a CXL component that implements the > > +CXL.mem protocol. It contains some amount of volatile memory, persistent > > memory, > > +or both. It is enumerated as a PCI device for configuration and passing > > +messages over an MMIO mailbox. Its contribution to the System Physical > > +Address space is handled via HDM (Host Managed Device Memory) decoders > > +that optionally define a device's contribution to an interleaved address > > +range across multiple devices underneath a host-bridge or interleaved > > +across host-bridges. > > + > > +Driver Infrastructure > > += > > + > > +This section covers the driver infrastructure for a CXL memory device. > > + > > +CXL Memor
Re: [PATCH v2 2/8] cxl/mem: Find device capabilities
On 21-02-10 15:07:59, Jonathan Cameron wrote: > On Wed, 10 Feb 2021 13:32:52 + > Jonathan Cameron wrote: > > > On Tue, 9 Feb 2021 16:02:53 -0800 > > Ben Widawsky wrote: > > > > > Provide enough functionality to utilize the mailbox of a memory device. > > > The mailbox is used to interact with the firmware running on the memory > > > device. The flow is proven with one implemented command, "identify". > > > Because the class code has already told the driver this is a memory > > > device and the identify command is mandatory. > > > > > > CXL devices contain an array of capabilities that describe the > > > interactions software can have with the device or firmware running on > > > the device. A CXL compliant device must implement the device status and > > > the mailbox capability. Additionally, a CXL compliant memory device must > > > implement the memory device capability. Each of the capabilities can > > > [will] provide an offset within the MMIO region for interacting with the > > > CXL device. > > > > > > The capabilities tell the driver how to find and map the register space > > > for CXL Memory Devices. The registers are required to utilize the CXL > > > spec defined mailbox interface. The spec outlines two mailboxes, primary > > > and secondary. The secondary mailbox is earmarked for system firmware, > > > and not handled in this driver. > > > > > > Primary mailboxes are capable of generating an interrupt when submitting > > > a background command. That implementation is saved for a later time. > > > > > > Link: https://www.computeexpresslink.org/download-the-specification > > > Signed-off-by: Ben Widawsky > > > Reviewed-by: Dan Williams > > > > Hi Ben, > > > > > > > +/** > > > + * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory device. > > > + * @cxlm: The CXL memory device to communicate with. > > > + * @mbox_cmd: Command to send to the memory device. > > > + * > > > + * Context: Any context. Expects mbox_lock to be held. > > > + * Return: -ETIMEDOUT if timeout occurred waiting for completion. 0 on > > > success. > > > + * Caller should check the return code in @mbox_cmd to make sure > > > it > > > + * succeeded. > > > > cxl_xfer_log() doesn't check mbox_cmd->return_code and for my test it > > currently > > enters an infinite loop as a result. I meant to fix that. > > > > I haven't checked other paths, but to my mind it is not a good idea to > > require > > two levels of error checking - the example here proves how easy it is to > > forget > > one. Demonstrably, you're correct. I think it would be good to have a kernel only mbox command that does the error checking though. Let me type something up and see how it looks. > > > > Now all I have to do is figure out why I'm getting an error in the first > > place! > > For reference this seems to be our old issue of arm64 memcpy_fromio() only > doing 8 byte > or 1 byte copies. The hack in QEMU to allow that to work, doesn't work. > Result is that 1 byte reads replicate across the register > (in this case instead of 001c I get 1c1c1c1c) > > For these particular registers, we are covered by the rules in 8.2 which says > that > a 1, 2, 4, 8 aligned reads of 64 bit registers etc are fine. > > So we should not have to care. This isn't true for the component registers > where > we need to guarantee 4 or 8 byte reads only. > > For this particular issue the mailbox_read_reg() function in the QEMU code > needs to handle the size 1 case and set min_access_size = 1 for > mailbox_ops. Logically it should also handle the 2 byte case I think, > but I'm not hitting that. > > Jonathan I think the latest QEMU patches should do the right thing (I have a v4 branch if you want to try it). If it doesn't, it'd be worth debugging. The memory accessors should split up or combine the reads/writes to whatever the emulation supports (4 or 8 only in this case). We can move this discussion to the QEMU list if it's not just a simple bug on my part. > > > > > Jonathan > > > > > > > > > + * > > > + * This is a generic form of the CXL mailbox send command, thus the only > > > I/O > > > + * operations used are cxl_read_mbox_reg(). Memory devices, and perhaps > > > other > > > + * types of CXL devices may have further information availab
Re: [PATCH v2 5/8] cxl/mem: Add a "RAW" send command
On 21-02-10 15:26:27, ariel.sib...@microchip.com wrote: > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > index c4ba3aa0a05d..08eaa8e52083 100644 > > --- a/drivers/cxl/Kconfig > > +++ b/drivers/cxl/Kconfig > > @@ -33,6 +33,24 @@ config CXL_MEM > > > > If unsure say 'm'. > > > > +config CXL_MEM_RAW_COMMANDS > > + bool "RAW Command Interface for Memory Devices" > > + depends on CXL_MEM > > + help > > + Enable CXL RAW command interface. > > + > > + The CXL driver ioctl interface may assign a kernel ioctl command > > + number for each specification defined opcode. At any given point > > in > > + time the number of opcodes that the specification defines and a > > device > > + may implement may exceed the kernel's set of associated ioctl > > function > > + numbers. The mismatch is either by omission, specification is too > > new, > > + or by design. When prototyping new hardware, or developing / > > debugging > > + the driver it is useful to be able to submit any possible command > > to > > + the hardware, even commands that may crash the kernel due to their > > + potential impact to memory currently in use by the kernel. > > + > > + If developing CXL hardware or the driver say Y, otherwise say N. > > Blocking RAW commands by default will prevent vendors from developing user > space tools that utilize vendor specific commands. Vendors of CXL.mem devices > should take ownership of ensuring any vendor defined commands that could cause > user data to be exposed or corrupted are disabled at the device level for > shipping configurations. Thanks for brining this up Ariel. If there is a recommendation on how to codify this, I would certainly like to know because the explanation will be long. --- The background: The enabling/disabling of the Kconfig option is driven by the distribution and/or system integrator. Even if we made the default 'y', nothing stops them from changing that. if you are using this driver in production and insist on using RAW commands, you are free to carry around a small patch to get rid of the WARN (it is a one-liner). To recap why this is in place - the driver owns the sanctity of the device and therefore a [large] part of the whole system. What we can do as driver writers is figure out the set of commands that are "safe" and allow those. Aside from being able to validate them, we're able to mediate them with other parallel operations that might conflict. We gain the ability to squint extra hard at bug reports. We provide a reason to try to use a well defined part of the spec. Realizing that only allowing that small set of commands in a rapidly growing ecosystem is not a welcoming API; we decided on RAW. Vendor commands can be one of two types: 1. Some functionality probably most vendors want. 2. Functionality that is really single vendor specific. Hopefully we can agree that the path for case #1 is to work with the consortium to standardize a command that does what is needed and that can eventually become part of UAPI. The situation is unfortunate, but temporary. If you won't be able to upgrade your kernel, patch out the WARN as above. The second situation is interesting and does need some more thought and discussion. --- I see 3 realistic options for truly vendor specific commands. 1. Tough noogies. Vendors aren't special and they shouldn't do that. 2. modparam to disable the WARN for specific devices (let the sysadmin decide) 3. Try to make them part of UAPI. The right answer to me is #1, but I also realize I live in the real world. #2 provides too much flexibility. Vendors will just do what they please and distros and/or integrators will be seen as hostile if they don't accommodate. I like #3, but I have a feeling not everyone will agree. My proposal for vendor specific commands is, if it's clear it's truly a unique command, allow adding it as part of UAPI (moving it out of RAW). I expect like 5 of these, ever. If we start getting multiple per vendor, we've failed. The infrastructure is already in place to allow doing this pretty easily. I think we'd have to draw up some guidelines (like adding test cases for the command) to allow these to come in. Anything with command effects is going to need extra scrutiny. In my opinion, as maintainers of the driver, we do owe the community an answer as to our direction for this. Dan, what is your thought? ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[PATCH v2 8/8] MAINTAINERS: Add maintainers of the CXL driver
Cc: Dan Williams Cc: Vishal Verma Cc: Ira Weiny Cc: Alison Schofield Signed-off-by: Ben Widawsky --- MAINTAINERS | 11 +++ 1 file changed, 11 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 6eff4f720c72..93c8694a8f04 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -,6 +,17 @@ M: Miguel Ojeda S: Maintained F: include/linux/compiler_attributes.h +COMPUTE EXPRESS LINK (CXL) +M: Alison Schofield +M: Vishal Verma +M: Ira Weiny +M: Ben Widawsky +M: Dan Williams +L: linux-...@vger.kernel.org +S: Maintained +F: drivers/cxl/ +F: include/uapi/linux/cxl_mem.h + CONEXANT ACCESSRUNNER USB DRIVER L: accessrunner-gene...@lists.sourceforge.net S: Orphan -- 2.30.0 ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
[PATCH v2 7/8] cxl/mem: Add set of informational commands
Add initial set of formal commands beyond basic identify and command
enumeration.
Of special note is the Get Log Command which is only specified to return
2 log types, CEL and VENDOR_DEBUG. Given that VENDOR_DEBUG is already a
large catch all for vendor specific information there is no known reason
for devices to be implementing other log types. Unknown log types are
included in the "vendor passthrough shenanigans" safety regime like raw
commands and blocked by default.
Up to this point there has been no reason to inspect payload data.
Given the need to check the log type add a new "validate_payload"
operation to define a generic mechanism to restrict / filter commands.
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams
---
drivers/cxl/mem.c| 55 +++-
include/uapi/linux/cxl_mem.h | 5
2 files changed, 59 insertions(+), 1 deletion(-)
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index e9aa6ca18d99..e8cc076b9f1b 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -44,12 +44,16 @@
enum opcode {
CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
+ CXL_MBOX_OP_GET_FW_INFO = 0x0200,
CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
CXL_MBOX_OP_GET_LOG = 0x0401,
CXL_MBOX_OP_IDENTIFY= 0x4000,
+ CXL_MBOX_OP_GET_PARTITION_INFO = 0x4100,
CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
+ CXL_MBOX_OP_GET_LSA = 0x4102,
CXL_MBOX_OP_SET_LSA = 0x4103,
+ CXL_MBOX_OP_GET_HEALTH_INFO = 0x4200,
CXL_MBOX_OP_SET_SHUTDOWN_STATE = 0x4204,
CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
@@ -118,6 +122,9 @@ static const uuid_t log_uuid[] = {
0xd6, 0x07, 0x19, 0x40, 0x3d, 0x86)
};
+static int validate_log_uuid(struct cxl_mem *cxlm, void __user *payload,
+size_t size);
+
/**
* struct cxl_mem_command - Driver representation of a memory device command
* @info: Command information as it exists for the UAPI
@@ -129,6 +136,10 @@ static const uuid_t log_uuid[] = {
* * %CXL_CMD_INTERNAL_FLAG_PSEUDO: This is a pseudo command which doesn't
have
*a direct mapping to hardware. They are implicitly always enabled.
*
+ * @validate_payload: A function called after the command is validated but
+ * before it's sent to the hardware. The primary purpose is to validate, or
+ * fixup the actual payload.
+ *
* The cxl_mem_command is the driver's internal representation of commands that
* are supported by the driver. Some of these commands may not be supported by
* the hardware. The driver will use @info to validate the fields passed in by
@@ -139,9 +150,12 @@ static const uuid_t log_uuid[] = {
struct cxl_mem_command {
struct cxl_command_info info;
enum opcode opcode;
+
+ int (*validate_payload)(struct cxl_mem *cxlm, void __user *payload,
+ size_t size);
};
-#define CXL_CMD(_id, _flags, sin, sout)
\
+#define CXL_CMD_VALIDATE(_id, _flags, sin, sout, v)
\
[CXL_MEM_COMMAND_ID_##_id] = { \
.info = { \
.id = CXL_MEM_COMMAND_ID_##_id,\
@@ -150,8 +164,12 @@ struct cxl_mem_command {
.size_out = sout, \
}, \
.opcode = CXL_MBOX_OP_##_id, \
+ .validate_payload = v, \
}
+#define CXL_CMD(_id, _flags, sin, sout)
\
+ CXL_CMD_VALIDATE(_id, _flags, sin, sout, NULL)
+
/*
* This table defines the supported mailbox commands for the driver. This table
* is made up of a UAPI structure. Non-negative values as parameters in the
@@ -164,6 +182,11 @@ static struct cxl_mem_command mem_commands[] = {
CXL_CMD(RAW, NONE, ~0, ~0),
#endif
CXL_CMD(GET_SUPPORTED_LOGS, NONE, 0, ~0),
+ CXL_CMD(GET_FW_INFO, NONE, 0, 0x50),
+ CXL_CMD(GET_PARTITION_INFO, NONE, 0, 0x20),
+ CXL_CMD(GET_LSA, NONE, 0x8, ~0),
+ CXL_CMD(GET_HEALTH_INFO, NONE, 0, 0x12),
+ CXL_CMD_VALIDATE(GET_LOG, NONE, 0x18, ~0, validate_log_uuid),
};
/*
@@ -492,6 +515,14 @@ static int handle_mailbox_cmd_from_user(struct cxl_memdev
*cxlmd,
mbox_cmd.payload_out = kvzalloc(cxlm->payload_size, GFP_KERNEL);
if (cmd->info.size_in) {
+ if (cmd->validate_payload) {
+
[PATCH v2 6/8] cxl/mem: Enable commands via CEL
CXL devices identified by the memory-device class code must implement
the Device Command Interface (described in 8.2.9 of the CXL 2.0 spec).
While the driver already maintains a list of commands it supports, there
is still a need to be able to distinguish between commands that the
driver knows about from commands that are optionally supported by the
hardware.
The Command Effects Log (CEL) is specified in the CXL 2.0 specification.
The CEL is one of two types of logs, the other being vendor specific.
They are distinguished in hardware/spec via UUID. The CEL is useful for
2 things:
1. Determine which optional commands are supported by the CXL device.
2. Enumerate any vendor specific commands
The CEL is used by the driver to determine which commands are available
in the hardware and therefore which commands userspace is allowed to
execute. The set of enabled commands might be a subset of commands which
are advertised in UAPI via CXL_MEM_SEND_COMMAND IOCTL.
The implementation leaves the statically defined table of commands and
supplements it with a bitmap to determine commands that are enabled.
This organization was chosen for the following reasons:
- Smaller memory footprint. Doesn't need a table per device.
- Reduce memory allocation complexity.
- Fixed command IDs to opcode mapping for all devices makes development
and debugging easier.
- Certain helpers are easily achievable, like cxl_for_each_cmd().
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams
---
drivers/cxl/cxl.h| 2 +
drivers/cxl/mem.c| 216 +++
include/uapi/linux/cxl_mem.h | 1 +
3 files changed, 219 insertions(+)
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index b3c56fa6e126..9a5e595abfa4 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -68,6 +68,7 @@ struct cxl_memdev;
*(CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
* @mbox_mutex: Mutex to synchronize mailbox access.
* @firmware_version: Firmware version for the memory device.
+ * @enabled_commands: Hardware commands found enabled in CEL.
* @pmem: Persistent memory capacity information.
* @ram: Volatile memory capacity information.
*/
@@ -83,6 +84,7 @@ struct cxl_mem {
size_t payload_size;
struct mutex mbox_mutex; /* Protects device mailbox and firmware */
char firmware_version[0x10];
+ unsigned long *enabled_cmds;
struct {
struct range range;
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 6d766a994dce..e9aa6ca18d99 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -45,6 +45,8 @@ enum opcode {
CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
+ CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
+ CXL_MBOX_OP_GET_LOG = 0x0401,
CXL_MBOX_OP_IDENTIFY= 0x4000,
CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
CXL_MBOX_OP_SET_LSA = 0x4103,
@@ -103,6 +105,19 @@ static DEFINE_IDA(cxl_memdev_ida);
static struct dentry *cxl_debugfs;
static bool raw_allow_all;
+enum {
+ CEL_UUID,
+ VENDOR_DEBUG_UUID
+};
+
+/* See CXL 2.0 Table 170. Get Log Input Payload */
+static const uuid_t log_uuid[] = {
+ [CEL_UUID] = UUID_INIT(0xda9c0b5, 0xbf41, 0x4b78, 0x8f, 0x79, 0x96,
+ 0xb1, 0x62, 0x3b, 0x3f, 0x17),
+ [VENDOR_DEBUG_UUID] = UUID_INIT(0xe1819d9, 0x11a9, 0x400c, 0x81, 0x1f,
+ 0xd6, 0x07, 0x19, 0x40, 0x3d, 0x86)
+};
+
/**
* struct cxl_mem_command - Driver representation of a memory device command
* @info: Command information as it exists for the UAPI
@@ -111,6 +126,8 @@ static bool raw_allow_all;
*
* * %CXL_CMD_FLAG_MANDATORY: Hardware must support this command. This flag is
*only used internally by the driver for sanity checking.
+ * * %CXL_CMD_INTERNAL_FLAG_PSEUDO: This is a pseudo command which doesn't
have
+ *a direct mapping to hardware. They are implicitly always enabled.
*
* The cxl_mem_command is the driver's internal representation of commands that
* are supported by the driver. Some of these commands may not be supported by
@@ -146,6 +163,7 @@ static struct cxl_mem_command mem_commands[] = {
#ifdef CONFIG_CXL_MEM_RAW_COMMANDS
CXL_CMD(RAW, NONE, ~0, ~0),
#endif
+ CXL_CMD(GET_SUPPORTED_LOGS, NONE, 0, ~0),
};
/*
@@ -627,6 +645,10 @@ static int cxl_validate_cmd_from_user(struct cxl_mem *cxlm,
c = &mem_commands[send_cmd->id];
info = &c->info;
+ /* Check that the command is enabled for hardware */
+ if (!test_bit(info->id, cxlm->enabled_cmds))
+ return -ENOTTY;
+
if (info->flags & CXL_MEM_COMMAND_FLAG_KERNEL)
return -EPERM;
@@ -869,6 +891,14 @@ static struct cxl_mem *cxl_mem_create(struct pci_dev
*
[PATCH v2 5/8] cxl/mem: Add a "RAW" send command
The CXL memory device send interface will have a number of supported
commands. The raw command is not such a command. Raw commands allow
userspace to send a specified opcode to the underlying hardware and
bypass all driver checks on the command. This is useful for a couple of
usecases, mainly:
1. Undocumented vendor specific hardware commands
2. Prototyping new hardware commands not yet supported by the driver
While this all sounds very powerful it comes with a couple of caveats:
1. Bug reports using raw commands will not get the same level of
attention as bug reports using supported commands (via taint).
2. Supported commands will be rejected by the RAW command.
With this comes new debugfs knob to allow full access to your toes with
your weapon of choice.
Cc: Ariel Sibley
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams
---
drivers/cxl/Kconfig | 18 +
drivers/cxl/mem.c| 125 ++-
include/uapi/linux/cxl_mem.h | 12 +++-
3 files changed, 152 insertions(+), 3 deletions(-)
diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
index c4ba3aa0a05d..08eaa8e52083 100644
--- a/drivers/cxl/Kconfig
+++ b/drivers/cxl/Kconfig
@@ -33,6 +33,24 @@ config CXL_MEM
If unsure say 'm'.
+config CXL_MEM_RAW_COMMANDS
+ bool "RAW Command Interface for Memory Devices"
+ depends on CXL_MEM
+ help
+ Enable CXL RAW command interface.
+
+ The CXL driver ioctl interface may assign a kernel ioctl command
+ number for each specification defined opcode. At any given point in
+ time the number of opcodes that the specification defines and a device
+ may implement may exceed the kernel's set of associated ioctl function
+ numbers. The mismatch is either by omission, specification is too new,
+ or by design. When prototyping new hardware, or developing / debugging
+ the driver it is useful to be able to submit any possible command to
+ the hardware, even commands that may crash the kernel due to their
+ potential impact to memory currently in use by the kernel.
+
+ If developing CXL hardware or the driver say Y, otherwise say N.
+
config CXL_MEM_INSECURE_DEBUG
bool "CXL.mem debugging"
depends on CXL_MEM
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index ce65630bb75e..6d766a994dce 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1,6 +1,8 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
#include
+#include
+#include
#include
#include
#include
@@ -41,7 +43,14 @@
enum opcode {
CXL_MBOX_OP_INVALID = 0x,
+ CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
+ CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
CXL_MBOX_OP_IDENTIFY= 0x4000,
+ CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
+ CXL_MBOX_OP_SET_LSA = 0x4103,
+ CXL_MBOX_OP_SET_SHUTDOWN_STATE = 0x4204,
+ CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
+ CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
CXL_MBOX_OP_MAX = 0x1
};
@@ -91,6 +100,8 @@ struct cxl_memdev {
static int cxl_mem_major;
static DEFINE_IDA(cxl_memdev_ida);
+static struct dentry *cxl_debugfs;
+static bool raw_allow_all;
/**
* struct cxl_mem_command - Driver representation of a memory device command
@@ -132,6 +143,49 @@ struct cxl_mem_command {
*/
static struct cxl_mem_command mem_commands[] = {
CXL_CMD(IDENTIFY, NONE, 0, 0x43),
+#ifdef CONFIG_CXL_MEM_RAW_COMMANDS
+ CXL_CMD(RAW, NONE, ~0, ~0),
+#endif
+};
+
+/*
+ * Commands that RAW doesn't permit. The rationale for each:
+ *
+ * CXL_MBOX_OP_ACTIVATE_FW: Firmware activation requires adjustment /
+ * coordination of transaction timeout values at the root bridge level.
+ *
+ * CXL_MBOX_OP_SET_PARTITION_INFO: The device memory map may change live
+ * and needs to be coordinated with HDM updates.
+ *
+ * CXL_MBOX_OP_SET_LSA: The label storage area may be cached by the
+ * driver and any writes from userspace invalidates those contents.
+ *
+ * CXL_MBOX_OP_SET_SHUTDOWN_STATE: Set shutdown state assumes no writes
+ * to the device after it is marked clean, userspace can not make that
+ * assertion.
+ *
+ * CXL_MBOX_OP_[GET_]SCAN_MEDIA: The kernel provides a native error list that
+ * is kept up to date with patrol notifications and error management.
+ */
+static u16 disabled_raw_commands[] = {
+ CXL_MBOX_OP_ACTIVATE_FW,
+ CXL_MBOX_OP_SET_PARTITION_INFO,
+ CXL_MBOX_OP_SET_LSA,
+ CXL_MBOX_OP_SET_SHUTDOWN_STATE,
+ CXL_MBOX_OP_SCAN_MEDIA,
+ CXL_MBOX_OP_GET_SCAN_MEDIA,
+};
+
+/*
+ * Command sets that RAW doesn't permit. All opcodes in this set are
+ * disabled because they pass plain text security payloads over the
+ * user/kernel boundary. This functionality is intended to be wra
[PATCH v2 4/8] cxl/mem: Add basic IOCTL interface
Add a straightforward IOCTL that provides a mechanism for userspace to
query the supported memory device commands. CXL commands as they appear
to userspace are described as part of the UAPI kerneldoc. The command
list returned via this IOCTL will contain the full set of commands that
the driver supports, however, some of those commands may not be
available for use by userspace.
Memory device commands first appear in the CXL 2.0 specification. They
are submitted through a mailbox mechanism specified also originally
specified in the CXL 2.0 specification.
The send command allows userspace to issue mailbox commands directly to
the hardware. The list of available commands to send are the output of
the query command. The driver verifies basic properties of the command
and possibly inspect the input (or output) payload to determine whether
or not the command is allowed (or might taint the kernel).
Reported-by: kernel test robot # bug in earlier revision
Signed-off-by: Ben Widawsky
Reviewed-by: Dan Williams
---
.clang-format | 1 +
.../userspace-api/ioctl/ioctl-number.rst | 1 +
drivers/cxl/mem.c | 291 +-
include/uapi/linux/cxl_mem.h | 152 +
4 files changed, 443 insertions(+), 2 deletions(-)
create mode 100644 include/uapi/linux/cxl_mem.h
diff --git a/.clang-format b/.clang-format
index 10dc5a9a61b3..3f11c8901b43 100644
--- a/.clang-format
+++ b/.clang-format
@@ -109,6 +109,7 @@ ForEachMacros:
- 'css_for_each_child'
- 'css_for_each_descendant_post'
- 'css_for_each_descendant_pre'
+ - 'cxl_for_each_cmd'
- 'device_for_each_child_node'
- 'dma_fence_chain_for_each'
- 'do_for_each_ftrace_op'
diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst
b/Documentation/userspace-api/ioctl/ioctl-number.rst
index a4c75a28c839..6eb8e634664d 100644
--- a/Documentation/userspace-api/ioctl/ioctl-number.rst
+++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
@@ -352,6 +352,7 @@ Code Seq#Include File
Comments
<mailto:michael.kl...@puffin.lb.shuttle.de>
0xCC 00-0F drivers/misc/ibmvmc.h pseries
VMC driver
0xCD 01 linux/reiserfs_fs.h
+0xCE 01-02 uapi/linux/cxl_mem.hCompute
Express Link Memory Devices
0xCF 02 fs/cifs/ioctl.c
0xDB 00-0F drivers/char/mwave/mwavepub.h
0xDD 00-3F ZFCP
device driver see drivers/s390/scsi/
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 8bbd2495e237..ce65630bb75e 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include
#include
#include
#include
@@ -39,6 +40,7 @@
#define CXL_MAILBOX_TIMEOUT_MS (2 * HZ)
enum opcode {
+ CXL_MBOX_OP_INVALID = 0x,
CXL_MBOX_OP_IDENTIFY= 0x4000,
CXL_MBOX_OP_MAX = 0x1
};
@@ -90,9 +92,57 @@ struct cxl_memdev {
static int cxl_mem_major;
static DEFINE_IDA(cxl_memdev_ida);
+/**
+ * struct cxl_mem_command - Driver representation of a memory device command
+ * @info: Command information as it exists for the UAPI
+ * @opcode: The actual bits used for the mailbox protocol
+ * @flags: Set of flags reflecting the state of the command.
+ *
+ * * %CXL_CMD_FLAG_MANDATORY: Hardware must support this command. This flag is
+ *only used internally by the driver for sanity checking.
+ *
+ * The cxl_mem_command is the driver's internal representation of commands that
+ * are supported by the driver. Some of these commands may not be supported by
+ * the hardware. The driver will use @info to validate the fields passed in by
+ * the user then submit the @opcode to the hardware.
+ *
+ * See struct cxl_command_info.
+ */
+struct cxl_mem_command {
+ struct cxl_command_info info;
+ enum opcode opcode;
+};
+
+#define CXL_CMD(_id, _flags, sin, sout)
\
+ [CXL_MEM_COMMAND_ID_##_id] = { \
+ .info = { \
+ .id = CXL_MEM_COMMAND_ID_##_id,\
+ .flags = CXL_MEM_COMMAND_FLAG_##_flags,\
+ .size_in = sin,\
+ .size_out = sout, \
+ }, \
+ .opcode = CXL_MBOX_OP_##_id, \
+ }
+
+/*
+ * This table defines t
[PATCH v2 3/8] cxl/mem: Register CXL memX devices
From: Dan Williams
Create the /sys/bus/cxl hierarchy to enumerate:
* Memory Devices (per-endpoint control devices)
* Memory Address Space Devices (platform address ranges with
interleaving, performance, and persistence attributes)
* Memory Regions (active provisioned memory from an address space device
that is in use as System RAM or delegated to libnvdimm as Persistent
Memory regions).
For now, only the per-endpoint control devices are registered on the
'cxl' bus. However, going forward it will provide a mechanism to
coordinate cross-device interleave.
Signed-off-by: Dan Williams
Signed-off-by: Ben Widawsky
---
Documentation/ABI/testing/sysfs-bus-cxl | 26 ++
.../driver-api/cxl/memory-devices.rst | 17 +
drivers/cxl/Makefile | 3 +
drivers/cxl/bus.c | 29 ++
drivers/cxl/cxl.h | 4 +
drivers/cxl/mem.c | 301 +-
6 files changed, 378 insertions(+), 2 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl
create mode 100644 drivers/cxl/bus.c
diff --git a/Documentation/ABI/testing/sysfs-bus-cxl
b/Documentation/ABI/testing/sysfs-bus-cxl
new file mode 100644
index ..2fe7490ad6a8
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-bus-cxl
@@ -0,0 +1,26 @@
+What: /sys/bus/cxl/devices/memX/firmware_version
+Date: December, 2020
+KernelVersion: v5.12
+Contact: linux-...@vger.kernel.org
+Description:
+ (RO) "FW Revision" string as reported by the Identify
+ Memory Device Output Payload in the CXL-2.0
+ specification.
+
+What: /sys/bus/cxl/devices/memX/ram/size
+Date: December, 2020
+KernelVersion: v5.12
+Contact: linux-...@vger.kernel.org
+Description:
+ (RO) "Volatile Only Capacity" as bytes. Represents the
+ identically named field in the Identify Memory Device Output
+ Payload in the CXL-2.0 specification.
+
+What: /sys/bus/cxl/devices/memX/pmem/size
+Date: December, 2020
+KernelVersion: v5.12
+Contact: linux-...@vger.kernel.org
+Description:
+ (RO) "Persistent Only Capacity" as bytes. Represents the
+ identically named field in the Identify Memory Device Output
+ Payload in the CXL-2.0 specification.
diff --git a/Documentation/driver-api/cxl/memory-devices.rst
b/Documentation/driver-api/cxl/memory-devices.rst
index 43177e700d62..1bad466f9167 100644
--- a/Documentation/driver-api/cxl/memory-devices.rst
+++ b/Documentation/driver-api/cxl/memory-devices.rst
@@ -27,3 +27,20 @@ CXL Memory Device
.. kernel-doc:: drivers/cxl/mem.c
:internal:
+
+CXL Bus
+---
+.. kernel-doc:: drivers/cxl/bus.c
+ :doc: cxl bus
+
+External Interfaces
+===
+
+CXL IOCTL Interface
+---
+
+.. kernel-doc:: include/uapi/linux/cxl_mem.h
+ :doc: UAPI
+
+.. kernel-doc:: include/uapi/linux/cxl_mem.h
+ :internal:
diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile
index 4a30f7c3fc4a..a314a1891f4d 100644
--- a/drivers/cxl/Makefile
+++ b/drivers/cxl/Makefile
@@ -1,4 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_CXL_BUS) += cxl_bus.o
obj-$(CONFIG_CXL_MEM) += cxl_mem.o
+ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
+cxl_bus-y := bus.o
cxl_mem-y := mem.o
diff --git a/drivers/cxl/bus.c b/drivers/cxl/bus.c
new file mode 100644
index ..58f74796d525
--- /dev/null
+++ b/drivers/cxl/bus.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include
+#include
+
+/**
+ * DOC: cxl bus
+ *
+ * The CXL bus provides namespace for control devices and a rendezvous
+ * point for cross-device interleave coordination.
+ */
+struct bus_type cxl_bus_type = {
+ .name = "cxl",
+};
+EXPORT_SYMBOL_GPL(cxl_bus_type);
+
+static __init int cxl_bus_init(void)
+{
+ return bus_register(&cxl_bus_type);
+}
+
+static void cxl_bus_exit(void)
+{
+ bus_unregister(&cxl_bus_type);
+}
+
+module_init(cxl_bus_init);
+module_exit(cxl_bus_exit);
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 745f5e0bfce3..b3c56fa6e126 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -3,6 +3,7 @@
#ifndef __CXL_H__
#define __CXL_H__
+#include
#include
#include
@@ -55,6 +56,7 @@
(FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \
CXLMDEV_RESET_NEEDED_NOT)
+struct cxl_memdev;
/**
* struct cxl_mem - A CXL memory device
* @pdev: The PCI device associated with this CXL device.
@@ -72,6 +74,7 @@
struct cxl_mem {
struct pci_dev *pdev;
void __iomem *regs;
+ struct cxl_memdev *cxlmd;
void __iomem *status_regs;
void __iomem *mbox_regs;
@@ -90,4 +93,5 @@ struct
[PATCH v2 1/8] cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints
From: Dan Williams The CXL.mem protocol allows a device to act as a provider of "System RAM" and/or "Persistent Memory" that is fully coherent as if the memory was attached to the typical CPU memory controller. With the CXL-2.0 specification a PCI endpoint can implement a "Type-3" device interface and give the operating system control over "Host Managed Device Memory". See section 2.3 Type 3 CXL Device. The memory range exported by the device may optionally be described by the platform firmware memory map, or by infrastructure like LIBNVDIMM to provision persistent memory capacity from one, or more, CXL.mem devices. A pre-requisite for Linux-managed memory-capacity provisioning is this cxl_mem driver that can speak the mailbox protocol defined in section 8.2.8.4 Mailbox Registers. For now just land the initial driver boiler-plate and Documentation/ infrastructure. Link: https://www.computeexpresslink.org/download-the-specification Cc: Jonathan Corbet Signed-off-by: Dan Williams Signed-off-by: Ben Widawsky Acked-by: David Rientjes (v1) --- Documentation/driver-api/cxl/index.rst| 12 .../driver-api/cxl/memory-devices.rst | 29 + Documentation/driver-api/index.rst| 1 + drivers/Kconfig | 1 + drivers/Makefile | 1 + drivers/cxl/Kconfig | 35 +++ drivers/cxl/Makefile | 4 ++ drivers/cxl/mem.c | 63 +++ drivers/cxl/pci.h | 18 ++ include/linux/pci_ids.h | 1 + 10 files changed, 165 insertions(+) create mode 100644 Documentation/driver-api/cxl/index.rst create mode 100644 Documentation/driver-api/cxl/memory-devices.rst create mode 100644 drivers/cxl/Kconfig create mode 100644 drivers/cxl/Makefile create mode 100644 drivers/cxl/mem.c create mode 100644 drivers/cxl/pci.h diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst new file mode 100644 index ..036e49553542 --- /dev/null +++ b/Documentation/driver-api/cxl/index.rst @@ -0,0 +1,12 @@ +.. SPDX-License-Identifier: GPL-2.0 + + +Compute Express Link + + +.. toctree:: + :maxdepth: 1 + + memory-devices + +.. only:: subproject and html diff --git a/Documentation/driver-api/cxl/memory-devices.rst b/Documentation/driver-api/cxl/memory-devices.rst new file mode 100644 index ..43177e700d62 --- /dev/null +++ b/Documentation/driver-api/cxl/memory-devices.rst @@ -0,0 +1,29 @@ +.. SPDX-License-Identifier: GPL-2.0 +.. include:: + +=== +Compute Express Link Memory Devices +=== + +A Compute Express Link Memory Device is a CXL component that implements the +CXL.mem protocol. It contains some amount of volatile memory, persistent memory, +or both. It is enumerated as a PCI device for configuration and passing +messages over an MMIO mailbox. Its contribution to the System Physical +Address space is handled via HDM (Host Managed Device Memory) decoders +that optionally define a device's contribution to an interleaved address +range across multiple devices underneath a host-bridge or interleaved +across host-bridges. + +Driver Infrastructure += + +This section covers the driver infrastructure for a CXL memory device. + +CXL Memory Device +- + +.. kernel-doc:: drivers/cxl/mem.c + :doc: cxl mem + +.. kernel-doc:: drivers/cxl/mem.c + :internal: diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst index 2456d0a97ed8..d246a18fd78f 100644 --- a/Documentation/driver-api/index.rst +++ b/Documentation/driver-api/index.rst @@ -35,6 +35,7 @@ available subsections can be seen below. usb/index firewire pci/index + cxl/index spi i2c ipmb diff --git a/drivers/Kconfig b/drivers/Kconfig index dcecc9f6e33f..62c753a73651 100644 --- a/drivers/Kconfig +++ b/drivers/Kconfig @@ -6,6 +6,7 @@ menu "Device Drivers" source "drivers/amba/Kconfig" source "drivers/eisa/Kconfig" source "drivers/pci/Kconfig" +source "drivers/cxl/Kconfig" source "drivers/pcmcia/Kconfig" source "drivers/rapidio/Kconfig" diff --git a/drivers/Makefile b/drivers/Makefile index fd11b9ac4cc3..678ea810410f 100644 --- a/drivers/Makefile +++ b/drivers/Makefile @@ -73,6 +73,7 @@ obj-$(CONFIG_NVM) += lightnvm/ obj-y += base/ block/ misc/ mfd/ nfc/ obj-$(CONFIG_LIBNVDIMM)+= nvdimm/ obj-$(CONFIG_DAX) += dax/ +obj-$(CONFIG_CXL_BUS) += cxl/ obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/ obj-$(CONFIG_NUBUS)+= nubus/ obj-y += macintosh/ diff --git a/drivers/cxl/
[PATCH v2 2/8] cxl/mem: Find device capabilities
Provide enough functionality to utilize the mailbox of a memory device. The mailbox is used to interact with the firmware running on the memory device. The flow is proven with one implemented command, "identify". Because the class code has already told the driver this is a memory device and the identify command is mandatory. CXL devices contain an array of capabilities that describe the interactions software can have with the device or firmware running on the device. A CXL compliant device must implement the device status and the mailbox capability. Additionally, a CXL compliant memory device must implement the memory device capability. Each of the capabilities can [will] provide an offset within the MMIO region for interacting with the CXL device. The capabilities tell the driver how to find and map the register space for CXL Memory Devices. The registers are required to utilize the CXL spec defined mailbox interface. The spec outlines two mailboxes, primary and secondary. The secondary mailbox is earmarked for system firmware, and not handled in this driver. Primary mailboxes are capable of generating an interrupt when submitting a background command. That implementation is saved for a later time. Link: https://www.computeexpresslink.org/download-the-specification Signed-off-by: Ben Widawsky Reviewed-by: Dan Williams --- drivers/cxl/Kconfig | 14 + drivers/cxl/cxl.h | 93 +++ drivers/cxl/mem.c | 511 +- drivers/cxl/pci.h | 13 + include/uapi/linux/pci_regs.h | 1 + 5 files changed, 630 insertions(+), 2 deletions(-) create mode 100644 drivers/cxl/cxl.h diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig index 9e80b311e928..c4ba3aa0a05d 100644 --- a/drivers/cxl/Kconfig +++ b/drivers/cxl/Kconfig @@ -32,4 +32,18 @@ config CXL_MEM Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification. If unsure say 'm'. + +config CXL_MEM_INSECURE_DEBUG + bool "CXL.mem debugging" + depends on CXL_MEM + help + Enable debug of all CXL command payloads. + + Some CXL devices and controllers support encryption and other + security features. The payloads for the commands that enable + those features may contain sensitive clear-text security + material. Disable debug of those command payloads by default. + If you are a kernel developer actively working on CXL + security enabling say Y, otherwise say N. + endif diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h new file mode 100644 index ..745f5e0bfce3 --- /dev/null +++ b/drivers/cxl/cxl.h @@ -0,0 +1,93 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2020 Intel Corporation. */ + +#ifndef __CXL_H__ +#define __CXL_H__ + +#include +#include +#include + +/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */ +#define CXLDEV_CAP_ARRAY_OFFSET 0x0 +#define CXLDEV_CAP_ARRAY_CAP_ID 0 +#define CXLDEV_CAP_ARRAY_ID_MASK GENMASK(15, 0) +#define CXLDEV_CAP_ARRAY_COUNT_MASK GENMASK(47, 32) +/* CXL 2.0 8.2.8.2.1 CXL Device Capabilities */ +#define CXLDEV_CAP_CAP_ID_DEVICE_STATUS 0x1 +#define CXLDEV_CAP_CAP_ID_PRIMARY_MAILBOX 0x2 +#define CXLDEV_CAP_CAP_ID_SECONDARY_MAILBOX 0x3 +#define CXLDEV_CAP_CAP_ID_MEMDEV 0x4000 + +/* CXL 2.0 8.2.8.4 Mailbox Registers */ +#define CXLDEV_MBOX_CAPS_OFFSET 0x00 +#define CXLDEV_MBOX_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0) +#define CXLDEV_MBOX_CTRL_OFFSET 0x04 +#define CXLDEV_MBOX_CTRL_DOORBELL BIT(0) +#define CXLDEV_MBOX_CMD_OFFSET 0x08 +#define CXLDEV_MBOX_CMD_COMMAND_OPCODE_MASK GENMASK(15, 0) +#define CXLDEV_MBOX_CMD_PAYLOAD_LENGTH_MASK GENMASK(36, 16) +#define CXLDEV_MBOX_STATUS_OFFSET 0x10 +#define CXLDEV_MBOX_STATUS_RET_CODE_MASK GENMASK(47, 32) +#define CXLDEV_MBOX_BG_CMD_STATUS_OFFSET 0x18 +#define CXLDEV_MBOX_PAYLOAD_OFFSET 0x20 + +/* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */ +#define CXLMDEV_STATUS_OFFSET 0x0 +#define CXLMDEV_DEV_FATAL BIT(0) +#define CXLMDEV_FW_HALT BIT(1) +#define CXLMDEV_STATUS_MEDIA_STATUS_MASK GENMASK(3, 2) +#define CXLMDEV_MS_NOT_READY 0 +#define CXLMDEV_MS_READY 1 +#define CXLMDEV_MS_ERROR 2 +#define CXLMDEV_MS_DISABLED 3 +#define CXLMDEV_READY(status) \ + (FIELD_GET(CXLMDEV_STATUS_MEDIA_STATUS_MASK, status) ==\ +CXLMDEV_MS_READY) +#define CXLMDEV_MBOX_IF_READY BIT(4) +#define CXLMDEV_RESET_NEEDED_MASK GENMASK(7, 5) +#define CXLMDEV_RESET_NEEDED_NOT 0 +#define CXLMDEV_RESET_NEEDED_COLD 1 +#define CXLMDEV_RESET_NEEDED_WARM 2 +#define CXLMDEV_RESET_NEEDED_HOT 3 +#define CXLMDEV_RESET_NEEDED_CXL 4 +#define CXLMDEV_RESET_NEEDED(status) \ + (FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \ +CXLMDEV_RESET_NEEDED_NOT) + +/** + * struct cxl_mem - A CXL me
[PATCH v2 0/8] CXL 2.0 Support
extra pgdat instances beyond what is enumerated in ACPI SRAT to accommodate hot-added CXL memory. Patches welcome, questions welcome as the development effort on the post v5.12 capabilities proceeds. ## Running in QEMU The incantation to get CXL support in QEMU [4] is considered unstable at this time. Future readers of this cover letter should verify if any changes are needed. For the novice QEMU user, the following can be copy/pasted into a working QEMU commandline. It is enough to make the simplest topology possible. The topology would consist of a single memory window, single type3 device, single root port, and single host bridge. +-+ | CXL PXB | | | | +---+ |<--+ | |CXL RP | | | +--+---+--+ v |+--+ || "window" | |+--+ v ^ +-+ | | CXL Type 3 | | | Device|<--+ +-+ // Memory backend for "window" -object memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M // Memory backend for LSA -object memory-backend-file,id=cxl-mem1-lsa,share,mem-path=cxl-mem1-lsa,size=1K // Host Bridge -device pxb-cxl id=cxl.0,bus=pcie.0,bus_nr=52,uid=0 len-window-base=1,window-base[0]=0x4c000 memdev[0]=cxl-mem1 // Single root port -device cxl rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0,memdev=cxl-mem1 // Single type3 device -device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M -device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M,lsa=cxl-mem1-lsa --- [1]: https://lore.kernel.org/linux-cxl/20210130002438.1872527-1-ben.widaw...@intel.com/ [2]: https://www.computeexpresslink.org/](https://www.computeexpresslink.org/) [3]: https://lore.kernel.org/qemu-devel/20210202005948.241655-1-ben.widaw...@intel.com/ [4]: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v4 [5]: https://github.com/pmem/ndctl/tree/cxl-2.0v2 --- Ben Widawsky (6): cxl/mem: Find device capabilities cxl/mem: Add basic IOCTL interface cxl/mem: Add a "RAW" send command cxl/mem: Enable commands via CEL cxl/mem: Add set of informational commands MAINTAINERS: Add maintainers of the CXL driver Dan Williams (2): cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints cxl/mem: Register CXL memX devices .clang-format |1 + Documentation/ABI/testing/sysfs-bus-cxl | 26 + Documentation/driver-api/cxl/index.rst| 12 + .../driver-api/cxl/memory-devices.rst | 46 + Documentation/driver-api/index.rst|1 + .../userspace-api/ioctl/ioctl-number.rst |1 + MAINTAINERS | 11 + drivers/Kconfig |1 + drivers/Makefile |1 + drivers/cxl/Kconfig | 67 + drivers/cxl/Makefile |7 + drivers/cxl/bus.c | 29 + drivers/cxl/cxl.h | 99 ++ drivers/cxl/mem.c | 1544 + drivers/cxl/pci.h | 31 + include/linux/pci_ids.h |1 + include/uapi/linux/cxl_mem.h | 168 ++ include/uapi/linux/pci_regs.h |1 + 18 files changed, 2047 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl create mode 100644 Documentation/driver-api/cxl/index.rst create mode 100644 Documentation/driver-api/cxl/memory-devices.rst create mode 100644 drivers/cxl/Kconfig create mode 100644 drivers/cxl/Makefile create mode 100644 drivers/cxl/bus.c create mode 100644 drivers/cxl/cxl.h create mode 100644 drivers/cxl/mem.c create mode 100644 drivers/cxl/pci.h create mode 100644 include/uapi/linux/cxl_mem.h Cc: linux-a...@vger.kernel.org Cc: linux-ker...@vger.kernel.org Cc: linux-nvdimm@lists.01.org Cc: linux-...@vger.kernel.org Cc: Bjorn Helgaas Cc: Chris Browy Cc: Christoph Hellwig Cc: Dan Williams Cc: David Hildenbrand Cc: David Rientjes Cc: Ira Weiny Cc: Jon Masters Cc: Jonathan Cameron Cc: Rafael Wysocki Cc: Randy Dunlap Cc: Vishal Verma Cc: "John Groves (jgroves)" Cc: "Kelley, Sean V" -- 2.30.0 ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 08/14] taint: add taint for direct hardware access
On 21-02-08 17:03:25, Dan Williams wrote: > On Mon, Feb 8, 2021 at 3:36 PM Dan Williams wrote: > > > > On Mon, Feb 8, 2021 at 2:09 PM Kees Cook wrote: > > > > > > On Mon, Feb 08, 2021 at 02:00:33PM -0800, Dan Williams wrote: > > > > [ add Jon Corbet as I'd expect him to be Cc'd on anything that > > > > generically touches Documentation/ like this, and add Kees as the last > > > > person who added a taint (tag you're it) ] > > > > > > > > Jon, Kees, are either of you willing to ack this concept? > > > > > > > > Top-posting to add more context for the below: > > > > > > > > This taint is proposed because it has implications for > > > > CONFIG_LOCK_DOWN_KERNEL among other things. These CXL devices > > > > implement memory like DDR would, but unlike DDR there are > > > > administrative / configuration commands that demand kernel > > > > coordination before they can be sent. The posture taken with this > > > > taint is "guilty until proven innocent" for commands that have yet to > > > > be explicitly allowed by the driver. This is different than NVME for > > > > example where an errant vendor-defined command could destroy data on > > > > the device, but there is no wider threat to system integrity. The > > > > taint allows a pressure release valve for any and all commands to be > > > > sent, but flagged with WARN_TAINT_ONCE if the driver has not > > > > explicitly enabled it on an allowed list of known-good / kernel > > > > coordinated commands. > > > > > > > > On Fri, Jan 29, 2021 at 4:25 PM Ben Widawsky > > > > wrote: > > > > > > > > > > For drivers that moderate access to the underlying hardware it is > > > > > sometimes desirable to allow userspace to bypass restrictions. Once > > > > > userspace has done this, the driver can no longer guarantee the > > > > > sanctity > > > > > of either the OS or the hardware. When in this state, it is helpful > > > > > for > > > > > kernel developers to be made aware (via this taint flag) of this fact > > > > > for subsequent bug reports. > > > > > > > > > > Example usage: > > > > > - Hardware xyzzy accepts 2 commands, waldo and fred. > > > > > - The xyzzy driver provides an interface for using waldo, but not > > > > > fred. > > > > > - quux is convinced they really need the fred command. > > > > > - xyzzy driver allows quux to frob hardware to initiate fred. > > > > > - kernel gets tainted. > > > > > - turns out fred command is borked, and scribbles over memory. > > > > > - developers laugh while closing quux's subsequent bug report. > > > > > > But a taint flag only lasts for the current boot. If this is a drive, it > > > could still be compromised after reboot. It sounds like this taint is > > > really only for ephemeral things? "vendor shenanigans" is a pretty giant > > > scope ... > > > > > > > That is true. This is more about preventing an ecosystem / cottage > > industry of tooling built around bypassing the kernel. So the kernel > > complains loudly and hopefully prevents vendor tooling from > > propagating and instead directs that development effort back to the > > native tooling. However for the rare "I know what I'm doing" cases, > > this tainted kernel bypass lets some experimentation and debug happen, > > but the kernel is transparent that when the capability ships in > > production it needs to be a native implementation. > > > > So it's less, "the system integrity is compromised" and more like > > "you're bypassing the development process that ensures sanity for CXL > > implementations that may take down a system if implemented > > incorrectly". For example, NVME reset is a non-invent, CXL reset can > > be like surprise removing DDR DIMM. > > > > Should this be more tightly scoped to CXL? I had hoped to use this in > > other places in LIBNVDIMM, but I'm ok to lose some generality for the > > specific concerns that make CXL devices different than other PCI > > endpoints. > > As I type this out it strikes me that plain WARN already does > TAINT_WARN and meets the spirit of what is trying to be achieved. > > Appreciate the skeptical eye Kees, we'll drop this one. So I think this is a good compromise for now. However, the point of this taint was that it is specifically called out what tainted the kernel. It'd be great to know when we have a bug report it was this specifically that was the issue. Rambling further I realize now that taint doesn't tell you which module tainted, which would be great here. That's actually what I'd like. End ramble. ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 08/14] taint: add taint for direct hardware access
On 21-02-08 14:09:19, Kees Cook wrote: > On Mon, Feb 08, 2021 at 02:00:33PM -0800, Dan Williams wrote: > > [ add Jon Corbet as I'd expect him to be Cc'd on anything that > > generically touches Documentation/ like this, and add Kees as the last > > person who added a taint (tag you're it) ] > > > > Jon, Kees, are either of you willing to ack this concept? > > > > Top-posting to add more context for the below: > > > > This taint is proposed because it has implications for > > CONFIG_LOCK_DOWN_KERNEL among other things. These CXL devices > > implement memory like DDR would, but unlike DDR there are > > administrative / configuration commands that demand kernel > > coordination before they can be sent. The posture taken with this > > taint is "guilty until proven innocent" for commands that have yet to > > be explicitly allowed by the driver. This is different than NVME for > > example where an errant vendor-defined command could destroy data on > > the device, but there is no wider threat to system integrity. The > > taint allows a pressure release valve for any and all commands to be > > sent, but flagged with WARN_TAINT_ONCE if the driver has not > > explicitly enabled it on an allowed list of known-good / kernel > > coordinated commands. > > > > On Fri, Jan 29, 2021 at 4:25 PM Ben Widawsky wrote: > > > > > > For drivers that moderate access to the underlying hardware it is > > > sometimes desirable to allow userspace to bypass restrictions. Once > > > userspace has done this, the driver can no longer guarantee the sanctity > > > of either the OS or the hardware. When in this state, it is helpful for > > > kernel developers to be made aware (via this taint flag) of this fact > > > for subsequent bug reports. > > > > > > Example usage: > > > - Hardware xyzzy accepts 2 commands, waldo and fred. > > > - The xyzzy driver provides an interface for using waldo, but not fred. > > > - quux is convinced they really need the fred command. > > > - xyzzy driver allows quux to frob hardware to initiate fred. > > > - kernel gets tainted. > > > - turns out fred command is borked, and scribbles over memory. > > > - developers laugh while closing quux's subsequent bug report. > > But a taint flag only lasts for the current boot. If this is a drive, it > could still be compromised after reboot. It sounds like this taint is > really only for ephemeral things? "vendor shenanigans" is a pretty giant > scope ... > > -Kees > Good point. Any suggestions? > > > > > > Signed-off-by: Ben Widawsky > > > --- > > > Documentation/admin-guide/sysctl/kernel.rst | 1 + > > > Documentation/admin-guide/tainted-kernels.rst | 6 +- > > > include/linux/kernel.h| 3 ++- > > > kernel/panic.c| 1 + > > > 4 files changed, 9 insertions(+), 2 deletions(-) > > > > > > diff --git a/Documentation/admin-guide/sysctl/kernel.rst > > > b/Documentation/admin-guide/sysctl/kernel.rst > > > index 1d56a6b73a4e..3e1eada53504 100644 > > > --- a/Documentation/admin-guide/sysctl/kernel.rst > > > +++ b/Documentation/admin-guide/sysctl/kernel.rst > > > @@ -1352,6 +1352,7 @@ ORed together. The letters are seen in "Tainted" > > > line of Oops reports. > > > 32768 `(K)` kernel has been live patched > > > 65536 `(X)` Auxiliary taint, defined and used by for distros > > > 131072 `(T)` The kernel was built with the struct randomization plugin > > > +262144 `(H)` The kernel has allowed vendor shenanigans > > > == = > > > == > > > > > > See :doc:`/admin-guide/tainted-kernels` for more information. > > > diff --git a/Documentation/admin-guide/tainted-kernels.rst > > > b/Documentation/admin-guide/tainted-kernels.rst > > > index ceeed7b0798d..ee2913316344 100644 > > > --- a/Documentation/admin-guide/tainted-kernels.rst > > > +++ b/Documentation/admin-guide/tainted-kernels.rst > > > @@ -74,7 +74,7 @@ a particular type of taint. It's best to leave that to > > > the aforementioned > > > script, but if you need something quick you can use this shell command > > > to check > > > which bits are set:: > > > > > > - $ for i in $(seq 18); do echo $(($i-1)) $(($(cat > > > /proc/sys/kernel/tainted)>>($i-1)&1));done > &
Re: [EXT] Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On 21-02-04 21:53:29, John Groves (jgroves) wrote: >Micron Confidential > > > >From: Dan Williams >Date: Monday, February 1, 2021 at 1:28 PM >To: Ben Widawsky >Cc: Konrad Rzeszutek Wilk , >linux-...@vger.kernel.org , Linux ACPI >, Linux Kernel Mailing List >, linux-nvdimm >, Linux PCI , >Bjorn Helgaas , Chris Browy >, Christoph Hellwig , Ira >Weiny , Jon Masters , Jonathan >Cameron , Rafael Wysocki >, Randy Dunlap , >Vishal Verma , daniel@alibaba-inc.com >, John Groves (jgroves) >, Kelley, Sean V >Subject: [EXT] Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox > >CAUTION: EXTERNAL EMAIL. Do not click links or open attachments unless >you recognize the sender and were expecting this message. >On Mon, Feb 1, 2021 at 11:13 AM Ben Widawsky >wrote: >> >> On 21-02-01 12:54:00, Konrad Rzeszutek Wilk wrote: >> > > +#define >cxl_doorbell_busy(cxlm) >\ >> > > + (cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET) >&\ >> > > +CXLDEV_MB_CTRL_DOORBELL) >> > > + >> > > +#define CXL_MAILBOX_TIMEOUT_US 2000 >> > >> > You been using the spec for the values. Is that number also from it >? >> > >> >> Yes it is. I'll add a comment with the spec reference. > > > >From section 8.2.8.4 in the CXL 2.0 spec: “The mailbox command timeout >is 2 seconds.” So this should be: > > >#define CXL_MAILBOX_TIMEOUT_US 200 > > >…right? 2000us is 2ms… > > Thanks. This was caught already in earlier review by David Rientjes It's renamed CXL_MAILBOX_TIMEOUT_MS 2000 ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 13/14] cxl/mem: Add limited Get Log command (0401h)
On 21-02-03 12:31:00, Dan Williams wrote:
> On Wed, Feb 3, 2021 at 10:16 AM Konrad Rzeszutek Wilk
> wrote:
> >
> > On Wed, Feb 03, 2021 at 09:16:10AM -0800, Ben Widawsky wrote:
> > > On 21-02-02 15:57:03, Dan Williams wrote:
> > > > On Tue, Feb 2, 2021 at 3:51 PM Ben Widawsky
> > > > wrote:
> > > > >
> > > > > On 21-02-01 13:28:48, Konrad Rzeszutek Wilk wrote:
> > > > > > On Fri, Jan 29, 2021 at 04:24:37PM -0800, Ben Widawsky wrote:
> > > > > > > The Get Log command returns the actual log entries that are
> > > > > > > advertised
> > > > > > > via the Get Supported Logs command (0400h). CXL device logs are
> > > > > > > selected
> > > > > > > by UUID which is part of the CXL spec. Because the driver tries to
> > > > > > > sanitize what is sent to hardware, there becomes a need to
> > > > > > > restrict the
> > > > > > > types of logs which can be accessed by userspace. For example, the
> > > > > > > vendor specific log might only be consumable by proprietary, or
> > > > > > > offline
> > > > > > > applications, and therefore a good candidate for userspace.
> > > > > > >
> > > > > > > The current driver infrastructure does allow basic validation for
> > > > > > > all
> > > > > > > commands, but doesn't inspect any of the payload data. Along with
> > > > > > > Get
> > > > > > > Log support comes new infrastructure to add a hook for payload
> > > > > > > validation. This infrastructure is used to filter out the CEL
> > > > > > > UUID,
> > > > > > > which the userspace driver doesn't have business knowing, and
> > > > > > > taints on
> > > > > > > invalid UUIDs being sent to hardware.
> > > > > >
> > > > > > Perhaps a better option is to reject invalid UUIDs?
> > > > > >
> > > > > > And if you really really want to use invalid UUIDs then:
> > > > > >
> > > > > > 1) Make that code wrapped in CONFIG_CXL_DEBUG_THIS_IS_GOING_TO..?
> > > > > >
> > > > > > 2) Wrap it with lockdown code so that you can't do this at all
> > > > > >when in LOCKDOWN_INTEGRITY or such?
> > > > > >
> > > > >
> > > > > The commit message needs update btw as CEL is allowed in the latest
> > > > > rev of the
> > > > > patches.
> > > > >
> > > > > We could potentially combine this with the now added (in a branch)
> > > > > CONFIG_RAW
> > > > > config option. Indeed I think that makes sense. Dan, thoughts?
> > > >
> > > > Yeah, unknown UUIDs blocking is the same risk as raw commands as a
> > > > vendor can trigger any behavior they want. A "CONFIG_RAW depends on
> > > > !CONFIG_INTEGRITY" policy sounds reasonable as well.
> > >
> > > What about LOCKDOWN_NONE though? I think we need something runtime for
> > > this.
> > >
> > > Can we summarize the CONFIG options here?
> > >
> > > CXL_MEM_INSECURE_DEBUG // no change
> > > CXL_MEM_RAW_COMMANDS // if !security_locked_down(LOCKDOWN_NONE)
> > >
> > > bool cxl_unsafe()
> >
> > Would it be better if this inverted? Aka cxl_safe()..
> > ?
> > > {
> > > #ifndef CXL_MEM_RAW_COMMANDS
>
> nit use IS_ENABLED() if this function lives in a C file, or provide
> whole alternate static inline versions in a header gated by ifdefs.
>
I had done this independently since... but agreed.
> > > return false;
> > > #else
> > > return !security_locked_down(LOCKDOWN_NONE);
> >
> > :thumbsup:
> >
> > (Naturally this would inverted if this was cxl_safe()).
> >
> >
> > > #endif
> > > }
> > >
> > > ---
> > >
> > > Did I get that right?
> >
> > :nods:
>
> Looks good which means it's time to bikeshed the naming. I'd call it
> cxl_raw_allowed(). As "safety" isn't the only reason for blocking raw,
> it's also to corral the userspace api. I.e. things like enforcing
> security passphrase material through the Linux keys api.
It actually got pushed into cxl_mem_raw_command_allowed()
static bool cxl_mem_raw_command_allowed(u16 opcode)
{
int i;
if (!IS_ENABLED(CONFIG_CXL_MEM_RAW_COMMANDS))
return false;
if (security_locked_down(LOCKDOWN_NONE))
return false;
if (raw_allow_all)
return true;
if (is_security_command(opcode))
return false;
for (i = 0; i < ARRAY_SIZE(disabled_raw_commands); i++)
if (disabled_raw_commands[i] == opcode)
return false;
return true;
}
That work for you?
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 03/14] cxl/mem: Find device capabilities
On 21-02-04 07:16:46, Christoph Hellwig wrote: > On Wed, Feb 03, 2021 at 01:23:31PM -0800, Dan Williams wrote: > > > I'd prefer to keep the helpers for now as I do find them helpful, and so > > > far > > > nobody else who has touched the code has complained. If you feel > > > strongly, I > > > will change it. > > > > After seeing the options, I think I'd prefer to not have to worry what > > extra magic is happening with cxl_read_mbox_reg32() > > > > cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CAPS_OFFSET); > > > > readl(cxlm->mbox_regs + CXLDEV_MB_CAPS_OFFSET); > > > > The latter is both shorter and more idiomatic. > > Same here. That being said I know some driver maintainers like > wrappers, my real main irk was the macro magic to generate them. I think the wrapper is often used as a means of trying to have cross OS compatibility to some degree. Just to be clear, that was *not* the purpose here. Stating I disagree for posterity, I'll begin reworking this code and it will be changed for v2. Thanks. Ben ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 03/14] cxl/mem: Find device capabilities
On 21-02-03 17:15:34, Christoph Hellwig wrote:
> On Tue, Feb 02, 2021 at 10:24:18AM -0800, Ben Widawsky wrote:
> > > > + /* Cap 4000h - CXL_CAP_CAP_ID_MEMDEV */
> > > > + struct {
> > > > + void __iomem *regs;
> > > > + } mem;
> > >
> > > This style looks massively obsfucated. For one the comments look like
> > > absolute gibberish, but also what is the point of all these anonymous
> > > structures?
> >
> > They're not anonymous, and their names are for the below register
> > functions. The
> > comments are connected spec reference 'Cap h' to definitions in cxl.h.
> > I can
> > articulate that if it helps.
>
> But why no simply a
>
> void __iomem *mem_regs;
>
> field vs the extra struct?
>
> > The register space for CXL devices is a bit weird since it's all subdivided
> > under 1 BAR for now. To clearly distinguish over the different subregions,
> > these
> > helpers exist. It's really easy to mess this up as the developer and I
> > actually
> > would disagree that it makes debugging quite a bit easier. It also gets more
> > convoluted when you add the other 2 BARs which also each have their own
> > subregions.
> >
> > For example. if my mailbox function does:
> > cxl_read_status_reg32(cxlm, CXLDEV_MB_CAPS_OFFSET);
> >
> > instead of:
> > cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CAPS_OFFSET);
> >
> > It's easier to spot than:
> > readl(cxlm->regs + cxlm->status_offset + CXLDEV_MB_CAPS_OFFSET)
>
> Well, what I think would be the most obvious is:
>
> readl(cxlm->status_regs + CXLDEV_MB_CAPS_OFFSET);
>
Right, so you wrote the buggy version. Should be.
readl(cxlm->mbox_regs + CXLDEV_MB_CAPS_OFFSET);
Admittedly, "MB" for mailbox isn't super obvious. I think you've convinced me to
rename these register definitions
s/MB/MBOX.
I'd prefer to keep the helpers for now as I do find them helpful, and so far
nobody else who has touched the code has complained. If you feel strongly, I
will change it.
> > > > + /* 8.2.8.4.3 */
> > >
> > >
> > >
> >
> > I had been trying to be consistent with 'CXL2.0 - ' in front of all spec
> > reference. I obviously missed this one.
>
> FYI, I generally find section names much easier to find than section
> numbers. Especially as the numbers change very frequently, some times
> even for very minor updates to the spec. E.g. in NVMe the numbers might
> even change from NVMe 1.X to NVMe 1.Xa because an errata had to add
> a clarification as its own section.
Why not both?
I ran into this in fact going from version 0.7 to 1.0 of the spec. I did call
out the spec version to address this, but you're right. Section names can change
too in theory.
/*
* CXL 2.0 8.2.8.4.3
* Mailbox Capabilities Register
*/
Too much?
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 13/14] cxl/mem: Add limited Get Log command (0401h)
On 21-02-02 15:57:03, Dan Williams wrote:
> On Tue, Feb 2, 2021 at 3:51 PM Ben Widawsky wrote:
> >
> > On 21-02-01 13:28:48, Konrad Rzeszutek Wilk wrote:
> > > On Fri, Jan 29, 2021 at 04:24:37PM -0800, Ben Widawsky wrote:
> > > > The Get Log command returns the actual log entries that are advertised
> > > > via the Get Supported Logs command (0400h). CXL device logs are selected
> > > > by UUID which is part of the CXL spec. Because the driver tries to
> > > > sanitize what is sent to hardware, there becomes a need to restrict the
> > > > types of logs which can be accessed by userspace. For example, the
> > > > vendor specific log might only be consumable by proprietary, or offline
> > > > applications, and therefore a good candidate for userspace.
> > > >
> > > > The current driver infrastructure does allow basic validation for all
> > > > commands, but doesn't inspect any of the payload data. Along with Get
> > > > Log support comes new infrastructure to add a hook for payload
> > > > validation. This infrastructure is used to filter out the CEL UUID,
> > > > which the userspace driver doesn't have business knowing, and taints on
> > > > invalid UUIDs being sent to hardware.
> > >
> > > Perhaps a better option is to reject invalid UUIDs?
> > >
> > > And if you really really want to use invalid UUIDs then:
> > >
> > > 1) Make that code wrapped in CONFIG_CXL_DEBUG_THIS_IS_GOING_TO..?
> > >
> > > 2) Wrap it with lockdown code so that you can't do this at all
> > >when in LOCKDOWN_INTEGRITY or such?
> > >
> >
> > The commit message needs update btw as CEL is allowed in the latest rev of
> > the
> > patches.
> >
> > We could potentially combine this with the now added (in a branch)
> > CONFIG_RAW
> > config option. Indeed I think that makes sense. Dan, thoughts?
>
> Yeah, unknown UUIDs blocking is the same risk as raw commands as a
> vendor can trigger any behavior they want. A "CONFIG_RAW depends on
> !CONFIG_INTEGRITY" policy sounds reasonable as well.
What about LOCKDOWN_NONE though? I think we need something runtime for this.
Can we summarize the CONFIG options here?
CXL_MEM_INSECURE_DEBUG // no change
CXL_MEM_RAW_COMMANDS // if !security_locked_down(LOCKDOWN_NONE)
bool cxl_unsafe()
{
#ifndef CXL_MEM_RAW_COMMANDS
return false;
#else
return !security_locked_down(LOCKDOWN_NONE);
#endif
}
---
Did I get that right?
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On 21-02-02 15:54:03, Dan Williams wrote: > On Tue, Feb 2, 2021 at 2:57 PM Ben Widawsky wrote: > > > > On 21-02-01 12:00:18, Dan Williams wrote: > > > On Sat, Jan 30, 2021 at 3:52 PM David Rientjes > > > wrote: > > > > > > > > On Fri, 29 Jan 2021, Ben Widawsky wrote: > > > > > > > > > Provide enough functionality to utilize the mailbox of a memory > > > > > device. > > > > > The mailbox is used to interact with the firmware running on the > > > > > memory > > > > > device. > > > > > > > > > > The CXL specification defines separate capabilities for the mailbox > > > > > and > > > > > the memory device. The mailbox interface has a doorbell to indicate > > > > > ready to accept commands and the memory device has a capability > > > > > register > > > > > that indicates the mailbox interface is ready. The expectation is that > > > > > the doorbell-ready is always later than the memory-device-indication > > > > > that the mailbox is ready. > > > > > > > > > > Create a function to handle sending a command, optionally with a > > > > > payload, to the memory device, polling on a result, and then > > > > > optionally > > > > > copying out the payload. The algorithm for doing this comes straight > > > > > out > > > > > of the CXL 2.0 specification. > > > > > > > > > > Primary mailboxes are capable of generating an interrupt when > > > > > submitting > > > > > a command in the background. That implementation is saved for a later > > > > > time. > > > > > > > > > > Secondary mailboxes aren't implemented at this time. > > > > > > > > > > The flow is proven with one implemented command, "identify". Because > > > > > the > > > > > class code has already told the driver this is a memory device and the > > > > > identify command is mandatory. > > > > > > > > > > Signed-off-by: Ben Widawsky > > > > > --- > > > > > drivers/cxl/Kconfig | 14 ++ > > > > > drivers/cxl/cxl.h | 39 + > > > > > drivers/cxl/mem.c | 342 > > > > > +++- > > > > > 3 files changed, 394 insertions(+), 1 deletion(-) > > > > > > > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > > > > index 3b66b46af8a0..fe591f74af96 100644 > > > > > --- a/drivers/cxl/Kconfig > > > > > +++ b/drivers/cxl/Kconfig > > > > > @@ -32,4 +32,18 @@ config CXL_MEM > > > > > Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification. > > > > > > > > > > If unsure say 'm'. > > > > > + > > > > > +config CXL_MEM_INSECURE_DEBUG > > > > > + bool "CXL.mem debugging" > > > > > + depends on CXL_MEM > > > > > + help > > > > > + Enable debug of all CXL command payloads. > > > > > + > > > > > + Some CXL devices and controllers support encryption and other > > > > > + security features. The payloads for the commands that enable > > > > > + those features may contain sensitive clear-text security > > > > > + material. Disable debug of those command payloads by default. > > > > > + If you are a kernel developer actively working on CXL > > > > > + security enabling say Y, otherwise say N. > > > > > > > > Not specific to this patch, but the reference to encryption made me > > > > curious about integrity: are all CXL.mem devices compatible with DIMP? > > > > Some? None? > > > > > > The encryption here is "device passphrase" similar to the NVDIMM > > > Security Management described here: > > > > > > https://pmem.io/documents/IntelOptanePMem_DSM_Interface-V2.0.pdf > > > > > > The LIBNVDIMM enabling wrapped this support with the Linux keys > > > interface which among other things enforces wrapping the clear text > > > passphrase with a Linux "trusted/encrypted" key. > > > > > > Additionally, the CXL.io interface optionally supports PCI IDE: > >
Re: [PATCH 13/14] cxl/mem: Add limited Get Log command (0401h)
On 21-02-01 13:28:48, Konrad Rzeszutek Wilk wrote:
> On Fri, Jan 29, 2021 at 04:24:37PM -0800, Ben Widawsky wrote:
> > The Get Log command returns the actual log entries that are advertised
> > via the Get Supported Logs command (0400h). CXL device logs are selected
> > by UUID which is part of the CXL spec. Because the driver tries to
> > sanitize what is sent to hardware, there becomes a need to restrict the
> > types of logs which can be accessed by userspace. For example, the
> > vendor specific log might only be consumable by proprietary, or offline
> > applications, and therefore a good candidate for userspace.
> >
> > The current driver infrastructure does allow basic validation for all
> > commands, but doesn't inspect any of the payload data. Along with Get
> > Log support comes new infrastructure to add a hook for payload
> > validation. This infrastructure is used to filter out the CEL UUID,
> > which the userspace driver doesn't have business knowing, and taints on
> > invalid UUIDs being sent to hardware.
>
> Perhaps a better option is to reject invalid UUIDs?
>
> And if you really really want to use invalid UUIDs then:
>
> 1) Make that code wrapped in CONFIG_CXL_DEBUG_THIS_IS_GOING_TO..?
>
> 2) Wrap it with lockdown code so that you can't do this at all
>when in LOCKDOWN_INTEGRITY or such?
>
The commit message needs update btw as CEL is allowed in the latest rev of the
patches.
We could potentially combine this with the now added (in a branch) CONFIG_RAW
config option. Indeed I think that makes sense. Dan, thoughts?
> >
> > Signed-off-by: Ben Widawsky
> > ---
> > drivers/cxl/mem.c| 42 +++-
> > include/uapi/linux/cxl_mem.h | 1 +
> > 2 files changed, 42 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > index b8ca6dff37b5..086268f1dd6c 100644
> > --- a/drivers/cxl/mem.c
> > +++ b/drivers/cxl/mem.c
> > @@ -119,6 +119,8 @@ static const uuid_t log_uuid[] = {
> > 0x07, 0x19, 0x40, 0x3d, 0x86)
> > };
> >
> > +static int validate_log_uuid(void __user *payload, size_t size);
> > +
> > /**
> > * struct cxl_mem_command - Driver representation of a memory device
> > command
> > * @info: Command information as it exists for the UAPI
> > @@ -132,6 +134,10 @@ static const uuid_t log_uuid[] = {
> > * * %CXL_CMD_INTERNAL_FLAG_PSEUDO: This is a pseudo command which
> > doesn't have
> > *a direct mapping to hardware. They are implicitly always enabled.
> > *
> > + * @validate_payload: A function called after the command is validated but
> > + * before it's sent to the hardware. The primary purpose is to validate, or
> > + * fixup the actual payload.
> > + *
> > * The cxl_mem_command is the driver's internal representation of commands
> > that
> > * are supported by the driver. Some of these commands may not be
> > supported by
> > * the hardware. The driver will use @info to validate the fields passed
> > in by
> > @@ -147,9 +153,11 @@ struct cxl_mem_command {
> > #define CXL_CMD_INTERNAL_FLAG_HIDDEN BIT(0)
> > #define CXL_CMD_INTERNAL_FLAG_MANDATORY BIT(1)
> > #define CXL_CMD_INTERNAL_FLAG_PSEUDO BIT(2)
> > +
> > + int (*validate_payload)(void __user *payload, size_t size);
> > };
> >
> > -#define CXL_CMD(_id, _flags, sin, sout, f)
> > \
> > +#define CXL_CMD_VALIDATE(_id, _flags, sin, sout, f, v)
> > \
> > [CXL_MEM_COMMAND_ID_##_id] = { \
> > .info = { \
> > .id = CXL_MEM_COMMAND_ID_##_id,\
> > @@ -159,8 +167,12 @@ struct cxl_mem_command {
> > }, \
> > .flags = CXL_CMD_INTERNAL_FLAG_##f,\
> > .opcode = CXL_MBOX_OP_##_id, \
> > + .validate_payload = v, \
> > }
> >
> > +#define CXL_CMD(_id, _flags, sin, sout, f)
> > \
> > + CXL_CMD_VALIDATE(_id, _flags, sin, sout, f, NULL)
> > +
> > /*
> > * This table defines the supported mailbox commands for the driver. This
> > table
> > * is made up of a UAPI structure. Non-negative values as parameters
Re: [PATCH 07/14] cxl/mem: Add send command
On 21-02-01 13:15:35, Konrad Rzeszutek Wilk wrote:
> > +/**
> > + * struct cxl_send_command - Send a command to a memory device.
> > + * @id: The command to send to the memory device. This must be one of the
> > + * commands returned by the query command.
> > + * @flags: Flags for the command (input).
> > + * @rsvd: Must be zero.
> > + * @retval: Return value from the memory device (output).
> > + * @size_in: Size of the payload to provide to the device (input).
> > + * @size_out: Size of the payload received from the device (input/output).
> > This
> > + * field is filled in by userspace to let the driver know how much
> > + * space was allocated for output. It is populated by the driver to
> > + * let userspace know how large the output payload actually was.
> > + * @in_payload: Pointer to memory for payload input (little endian order).
> > + * @out_payload: Pointer to memory for payload output (little endian
> > order).
> > + *
> > + * Mechanism for userspace to send a command to the hardware for
> > processing. The
> > + * driver will do basic validation on the command sizes. In some cases
> > even the
> > + * payload may be introspected. Userspace is required to allocate large
> > + * enough buffers for size_out which can be variable length in certain
> > + * situations.
> > + */
> I think (and this would help if you ran `pahole` on this structure) has
> some gaps in it:
>
> > +struct cxl_send_command {
> > + __u32 id;
> > + __u32 flags;
> > + __u32 rsvd;
> > + __u32 retval;
> > +
> > + struct {
> > + __s32 size_in;
>
> Here..Maybe just add:
>
> __u32 rsv_2;
> > + __u64 in_payload;
> > + };
> > +
> > + struct {
> > + __s32 size_out;
>
> And here. Maybe just add:
> __u32 rsv_2;
> > + __u64 out_payload;
> > + };
> > +};
>
> Perhaps to prepare for the future where this may need to be expanded, you
> could add a size at the start of the structure, and
> maybe what version of structure it is?
>
> Maybe for all the new structs you are adding?
Thanks for catching the holes. It broke somewhere in the earlier RFC changes.
I don't think we need to size or version. Reserved fields are good enough near
term future proofing and if we get to a point where the command is woefully
incompetent, I think it'd be time to just make cxl_send_command2.
Generally, I think cxl_send_command is fairly future proof because it's so
simple. As you get more complex, you might need better mechanisms, like deferred
command completion for example. It's unclear to me whether we'll get to that
point though, and if we do, I think a new command is warranted.
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On 21-02-01 12:00:18, Dan Williams wrote: > On Sat, Jan 30, 2021 at 3:52 PM David Rientjes wrote: > > > > On Fri, 29 Jan 2021, Ben Widawsky wrote: > > > > > Provide enough functionality to utilize the mailbox of a memory device. > > > The mailbox is used to interact with the firmware running on the memory > > > device. > > > > > > The CXL specification defines separate capabilities for the mailbox and > > > the memory device. The mailbox interface has a doorbell to indicate > > > ready to accept commands and the memory device has a capability register > > > that indicates the mailbox interface is ready. The expectation is that > > > the doorbell-ready is always later than the memory-device-indication > > > that the mailbox is ready. > > > > > > Create a function to handle sending a command, optionally with a > > > payload, to the memory device, polling on a result, and then optionally > > > copying out the payload. The algorithm for doing this comes straight out > > > of the CXL 2.0 specification. > > > > > > Primary mailboxes are capable of generating an interrupt when submitting > > > a command in the background. That implementation is saved for a later > > > time. > > > > > > Secondary mailboxes aren't implemented at this time. > > > > > > The flow is proven with one implemented command, "identify". Because the > > > class code has already told the driver this is a memory device and the > > > identify command is mandatory. > > > > > > Signed-off-by: Ben Widawsky > > > --- > > > drivers/cxl/Kconfig | 14 ++ > > > drivers/cxl/cxl.h | 39 + > > > drivers/cxl/mem.c | 342 +++- > > > 3 files changed, 394 insertions(+), 1 deletion(-) > > > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig > > > index 3b66b46af8a0..fe591f74af96 100644 > > > --- a/drivers/cxl/Kconfig > > > +++ b/drivers/cxl/Kconfig > > > @@ -32,4 +32,18 @@ config CXL_MEM > > > Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification. > > > > > > If unsure say 'm'. > > > + > > > +config CXL_MEM_INSECURE_DEBUG > > > + bool "CXL.mem debugging" > > > + depends on CXL_MEM > > > + help > > > + Enable debug of all CXL command payloads. > > > + > > > + Some CXL devices and controllers support encryption and other > > > + security features. The payloads for the commands that enable > > > + those features may contain sensitive clear-text security > > > + material. Disable debug of those command payloads by default. > > > + If you are a kernel developer actively working on CXL > > > + security enabling say Y, otherwise say N. > > > > Not specific to this patch, but the reference to encryption made me > > curious about integrity: are all CXL.mem devices compatible with DIMP? > > Some? None? > > The encryption here is "device passphrase" similar to the NVDIMM > Security Management described here: > > https://pmem.io/documents/IntelOptanePMem_DSM_Interface-V2.0.pdf > > The LIBNVDIMM enabling wrapped this support with the Linux keys > interface which among other things enforces wrapping the clear text > passphrase with a Linux "trusted/encrypted" key. > > Additionally, the CXL.io interface optionally supports PCI IDE: > > https://www.intel.com/content/dam/www/public/us/en/documents/reference-guides/pcie-device-security-enhancements.pdf > > I'm otherwise not familiar with the DIMP acronym? > > > > + > > > endif > > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > > > index a3da7f8050c4..df3d97154b63 100644 > > > --- a/drivers/cxl/cxl.h > > > +++ b/drivers/cxl/cxl.h > > > @@ -31,9 +31,36 @@ > > > #define CXLDEV_MB_CAPS_OFFSET 0x00 > > > #define CXLDEV_MB_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0) > > > #define CXLDEV_MB_CTRL_OFFSET 0x04 > > > +#define CXLDEV_MB_CTRL_DOORBELL BIT(0) > > > #define CXLDEV_MB_CMD_OFFSET 0x08 > > > +#define CXLDEV_MB_CMD_COMMAND_OPCODE_MASK GENMASK(15, 0) > > > +#define CXLDEV_MB_CMD_PAYLOAD_LENGTH_MASK GENMASK(36, 16) > > > #define CXLDEV_MB_STATUS_OFFSET 0x10 > > > +#define CXLDEV_MB_STATUS_RET_CODE_MASK GENMASK(47, 32) > > > #define CXLDEV_MB_BG_CMD_STATUS_OFFSET 0x18 > >
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On 21-01-30 15:51:57, David Rientjes wrote:
> On Fri, 29 Jan 2021, Ben Widawsky wrote:
>
[snip]
> > +/**
> > + * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory device.
> > + * @cxlm: The CXL memory device to communicate with.
> > + * @mbox_cmd: Command to send to the memory device.
> > + *
> > + * Context: Any context. Expects mbox_lock to be held.
> > + * Return: -ETIMEDOUT if timeout occurred waiting for completion. 0 on
> > success.
> > + * Caller should check the return code in @mbox_cmd to make sure it
> > + * succeeded.
> > + *
> > + * This is a generic form of the CXL mailbox send command, thus the only
> > I/O
> > + * operations used are cxl_read_mbox_reg(). Memory devices, and perhaps
> > other
> > + * types of CXL devices may have further information available upon error
> > + * conditions.
> > + *
> > + * The CXL spec allows for up to two mailboxes. The intention is for the
> > primary
> > + * mailbox to be OS controlled and the secondary mailbox to be used by
> > system
> > + * firmware. This allows the OS and firmware to communicate with the
> > device and
> > + * not need to coordinate with each other. The driver only uses the primary
> > + * mailbox.
> > + */
> > +static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
> > +struct mbox_cmd *mbox_cmd)
> > +{
> > + void __iomem *payload = cxlm->mbox.regs + CXLDEV_MB_PAYLOAD_OFFSET;
>
> Do you need to verify the payload is non-empty per 8.2.8.4?
>
What do you mean exactly? Emptiness or lack thereof is what determines the size
parameter of the mailbox interface (if we want to input data, we need to write
size, if we got output data, we have to read size bytes out of the payload
registers).
Perhaps I miss the point though, could you elaborate?
[snip]
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 06/14] cxl/mem: Add basic IOCTL interface
On 21-02-02 18:15:05, Christoph Hellwig wrote:
> > +#if defined(__cplusplus)
> > +extern "C" {
> > +#endif
>
> This has no business in a kernel header.
This was copypasta from DRM headers (which as you're probably aware wasn't
always part of the kernel)... It's my mistake and I will get rid of it.
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 02/14] cxl/mem: Map memory device registers
On 21-02-02 18:04:41, Christoph Hellwig wrote:
> Any reason not to merge a bunch of patches? Both this one and
> the previous one are rather useless on their own, making review
> harder than necessary.
>
As this is an initial driver, there's obviously no functional/regression testing
value in separating the patches.
This was the way we brought up the driver and how we verified functionality
incrementally. I see value in both capturing that in the history, as well as
making review a bit easier (which apparently failed for you).
> > + * cxl_mem_create() - Create a new &struct cxl_mem.
> > + * @pdev: The pci device associated with the new &struct cxl_mem.
> > + * @reg_lo: Lower 32b of the register locator
> > + * @reg_hi: Upper 32b of the register locator.
> > + *
> > + * Return: The new &struct cxl_mem on success, NULL on failure.
> > + *
> > + * Map the BAR for a CXL memory device. This BAR has the memory device's
> > + * registers for the device as specified in CXL specification.
> > + */
>
> A lot of text with almost no value over just reading the function.
> What's that fetish with kerneldoc comments for trivial static functions?
>
> > + reg_type =
> > + (reg_lo >> CXL_REGLOC_RBI_SHIFT) & CXL_REGLOC_RBI_MASK;
>
> OTOH this screams for a helper that would make the code a lot more
> self documenting.
>
I agree, I'll change this.
> > + if (reg_type == CXL_REGLOC_RBI_MEMDEV) {
> > + rc = 0;
> > + cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
> > + if (!cxlm)
> > + rc = -ENODEV;
> > + break;
>
> And given that we're going to grow more types eventually, why not start
> out with a switch here? Also why return the structure when nothing
> uses it?
We've (Intel) already started working on the libnvdimm integration which does
change this around a bit. In order to go with what's best tested though, I've
chosen to use this as is for merge. Many different people not just in Intel
have tested these codepaths. The resulting code moves the check on register
type out of this function entirely.
If you'd like me to make it a switch, I can, but it's going to be extracted
later anyway.
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 03/14] cxl/mem: Find device capabilities
On 21-02-02 18:10:16, Christoph Hellwig wrote:
> On Fri, Jan 29, 2021 at 04:24:27PM -0800, Ben Widawsky wrote:
> > #ifndef __CXL_H__
> > #define __CXL_H__
> >
> > +#include
> > +#include
> > +#include
> > +
> > +#define CXL_SET_FIELD(value, field)
> > \
> > + ({ \
> > + WARN_ON(!FIELD_FIT(field##_MASK, value)); \
> > + FIELD_PREP(field##_MASK, value); \
> > + })
> > +
> > +#define CXL_GET_FIELD(word, field) FIELD_GET(field##_MASK, word)
>
> This looks like some massive obsfucation. What is the intent
> here?
>
I will drop these. I don't recall why I did this to be honest.
> > + /* Cap 0001h - CXL_CAP_CAP_ID_DEVICE_STATUS */
> > + struct {
> > + void __iomem *regs;
> > + } status;
> > +
> > + /* Cap 0002h - CXL_CAP_CAP_ID_PRIMARY_MAILBOX */
> > + struct {
> > + void __iomem *regs;
> > + size_t payload_size;
> > + } mbox;
> > +
> > + /* Cap 4000h - CXL_CAP_CAP_ID_MEMDEV */
> > + struct {
> > + void __iomem *regs;
> > + } mem;
>
> This style looks massively obsfucated. For one the comments look like
> absolute gibberish, but also what is the point of all these anonymous
> structures?
They're not anonymous, and their names are for the below register functions. The
comments are connected spec reference 'Cap h' to definitions in cxl.h. I can
articulate that if it helps.
>
> > +#define cxl_reg(type)
> > \
> > + static inline void cxl_write_##type##_reg32(struct cxl_mem *cxlm, \
> > + u32 reg, u32 value)\
> > + { \
> > + void __iomem *reg_addr = cxlm->type.regs; \
> > + writel(value, reg_addr + reg); \
> > + } \
> > + static inline void cxl_write_##type##_reg64(struct cxl_mem *cxlm, \
> > + u32 reg, u64 value)\
> > + { \
> > + void __iomem *reg_addr = cxlm->type.regs; \
> > + writeq(value, reg_addr + reg); \
> > + } \
>
> What is the value add of all this obsfucation over the trivial
> calls to the write*/read* functions, possible with a locally
> declarate "void __iomem *" variable in the callers like in all
> normall drivers? Except for making the life of the poor soul trying
> to debug this code some time in the future really hard, of course.
>
The register space for CXL devices is a bit weird since it's all subdivided
under 1 BAR for now. To clearly distinguish over the different subregions, these
helpers exist. It's really easy to mess this up as the developer and I actually
would disagree that it makes debugging quite a bit easier. It also gets more
convoluted when you add the other 2 BARs which also each have their own
subregions.
For example. if my mailbox function does:
cxl_read_status_reg32(cxlm, CXLDEV_MB_CAPS_OFFSET);
instead of:
cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CAPS_OFFSET);
It's easier to spot than:
readl(cxlm->regs + cxlm->status_offset + CXLDEV_MB_CAPS_OFFSET)
> > + /* 8.2.8.4.3 */
>
>
>
I had been trying to be consistent with 'CXL2.0 - ' in front of all spec
reference. I obviously missed this one.
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 03/14] cxl/mem: Find device capabilities
On 21-02-01 15:58:09, David Rientjes wrote: > On Mon, 1 Feb 2021, Ben Widawsky wrote: > > > > I haven't seen the update to 8.2.8.4.5 to know yet :) > > > > > > You make a good point of at least being able to interact with the driver. > > > > > > I think you could argue that if the driver binds, then the payload size > > > is > > > accepted, in which case it would be strange to get an EINVAL when using > > > the ioctl with anything >1MB. > > > > > > Concern was that if we mask off the reserved bits from the command > > > register that we could be masking part of the payload size that is being > > > passed if the accepted max is >1MB. Idea was to avoid any possibility of > > > this inconsistency. If this is being checked for ioctl, seems like it's > > > checking reserved bits. > > > > > > But maybe I should just wait for the spec update. > > > > Well, I wouldn't hold your breath (it would be an errata in this case > > anyway). > > My preference would be to just allow allow mailbox payload size to be 2^31 > > and > > not deal with this. > > > > My question was how strongly do you feel it's an error that should prevent > > binding. > > > > I don't have an objection to binding, but doesn't this require that the > check in cxl_validate_cmd_from_user() guarantees send_cmd->size_in cannot > be greater than 1MB? You're correct. I'd need to add: cxlm->mbox.payload_size = min_t(size_t, 1 << CXL_GET_FIELD(cap, CXLDEV_MB_CAP_PAYLOAD_SIZE), 1<<20) ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 03/14] cxl/mem: Find device capabilities
On 21-02-01 15:09:45, David Rientjes wrote: > On Mon, 1 Feb 2021, Ben Widawsky wrote: > > > > I think that's what 8.2.8.4.3 says, no? And then 8.2.8.4.5 says you > > > can use up to Payload Size. That's why my recommendation was to enforce > > > this in cxl_mem_setup_mailbox() up front. > > > > Yeah. I asked our spec people to update 8.2.8.4.5 to make it clearer. I'd > > argue > > the intent is how you describe it, but the implementation isn't. > > > > My argument was silly anyway because if you specify greater than 1M as your > > payload, you will get EINVAL at the ioctl. > > > > The value of how it works today is the driver will at least bind and allow > > you > > to interact with it. > > > > How strongly do you feel about this? > > > > I haven't seen the update to 8.2.8.4.5 to know yet :) > > You make a good point of at least being able to interact with the driver. > I think you could argue that if the driver binds, then the payload size is > accepted, in which case it would be strange to get an EINVAL when using > the ioctl with anything >1MB. > > Concern was that if we mask off the reserved bits from the command > register that we could be masking part of the payload size that is being > passed if the accepted max is >1MB. Idea was to avoid any possibility of > this inconsistency. If this is being checked for ioctl, seems like it's > checking reserved bits. > > But maybe I should just wait for the spec update. Well, I wouldn't hold your breath (it would be an errata in this case anyway). My preference would be to just allow allow mailbox payload size to be 2^31 and not deal with this. My question was how strongly do you feel it's an error that should prevent binding. ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 03/14] cxl/mem: Find device capabilities
On 21-02-01 14:45:00, David Rientjes wrote:
> On Mon, 1 Feb 2021, Ben Widawsky wrote:
>
> > > > > > > > > +static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm)
> > > > > > > > > +{
> > > > > > > > > + const int cap = cxl_read_mbox_reg32(cxlm,
> > > > > > > > > CXLDEV_MB_CAPS_OFFSET);
> > > > > > > > > +
> > > > > > > > > + cxlm->mbox.payload_size =
> > > > > > > > > + 1 << CXL_GET_FIELD(cap,
> > > > > > > > > CXLDEV_MB_CAP_PAYLOAD_SIZE);
> > > > > > > > > +
> > > > > > > > > + /* 8.2.8.4.3 */
> > > > > > > > > + if (cxlm->mbox.payload_size < 256) {
> > > > > > > > > + dev_err(&cxlm->pdev->dev, "Mailbox is too small
> > > > > > > > > (%zub)",
> > > > > > > > > + cxlm->mbox.payload_size);
> > > > > > > > > + return -ENXIO;
> > > > > > > > > + }
> > > > > > > >
> > > > > > > > Any reason not to check cxlm->mbox.payload_size > (1 << 20) as
> > > > > > > > well and
> > > > > > > > return ENXIO if true?
> > > > > > >
> > > > > > > If some crazy vendor wanted to ship a mailbox larger than 1M, why
> > > > > > > should the
> > > > > > > driver not allow it?
> > > > > > >
> > > > > >
> > > > > > Because the spec disallows it :)
> > > > >
> > > > > I don't see it being the driver's responsibility to enforce spec
> > > > > correctness
> > > > > though. In certain cases, I need to use the spec, like I have to pick
> > > > > /some/
> > > > > mailbox timeout. For other cases...
> > > > >
> > > > > I'm not too familiar with what other similar drivers may or may not
> > > > > do in
> > > > > situations like this. The current 256 limit is mostly a reflection of
> > > > > that being
> > > > > too small to even support advertised mandatory commands. So things
> > > > > can't work in
> > > > > that scenario, but things can work if they have a larger register
> > > > > size (so long
> > > > > as the BAR advertises enough space).
> > > > >
> > > >
> > > > I don't think things can work above 1MB, either, though. Section
> > > > 8.2.8.4.5 specifies 20 bits to define the payload length, if this is
> > > > larger than cxlm->mbox.payload_size it would venture into the reserved
> > > > bits of the command register.
> > > >
> > > > So is the idea to allow cxl_mem_setup_mailbox() to succeed with a
> > > > payload
> > > > size > 1MB and then only check 20 bits for the command register?
> > >
> > > So it's probably a spec bug, but actually the payload size is 21 bits...
> > > I'll
> > > check if that was a mistake.
> >
> > Well I guess they wanted to be able to specify 1M exactly... Spec should
> > probably say you can't go over 1M
> >
>
> I think that's what 8.2.8.4.3 says, no? And then 8.2.8.4.5 says you
> can use up to Payload Size. That's why my recommendation was to enforce
> this in cxl_mem_setup_mailbox() up front.
Yeah. I asked our spec people to update 8.2.8.4.5 to make it clearer. I'd argue
the intent is how you describe it, but the implementation isn't.
My argument was silly anyway because if you specify greater than 1M as your
payload, you will get EINVAL at the ioctl.
The value of how it works today is the driver will at least bind and allow you
to interact with it.
How strongly do you feel about this?
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 03/14] cxl/mem: Find device capabilities
On 21-02-01 14:28:59, Ben Widawsky wrote:
> On 21-02-01 14:23:47, David Rientjes wrote:
> > On Mon, 1 Feb 2021, Ben Widawsky wrote:
> >
> > > > > > > +static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm)
> > > > > > > +{
> > > > > > > + const int cap = cxl_read_mbox_reg32(cxlm,
> > > > > > > CXLDEV_MB_CAPS_OFFSET);
> > > > > > > +
> > > > > > > + cxlm->mbox.payload_size =
> > > > > > > + 1 << CXL_GET_FIELD(cap, CXLDEV_MB_CAP_PAYLOAD_SIZE);
> > > > > > > +
> > > > > > > + /* 8.2.8.4.3 */
> > > > > > > + if (cxlm->mbox.payload_size < 256) {
> > > > > > > + dev_err(&cxlm->pdev->dev, "Mailbox is too small (%zub)",
> > > > > > > + cxlm->mbox.payload_size);
> > > > > > > + return -ENXIO;
> > > > > > > + }
> > > > > >
> > > > > > Any reason not to check cxlm->mbox.payload_size > (1 << 20) as well
> > > > > > and
> > > > > > return ENXIO if true?
> > > > >
> > > > > If some crazy vendor wanted to ship a mailbox larger than 1M, why
> > > > > should the
> > > > > driver not allow it?
> > > > >
> > > >
> > > > Because the spec disallows it :)
> > >
> > > I don't see it being the driver's responsibility to enforce spec
> > > correctness
> > > though. In certain cases, I need to use the spec, like I have to pick
> > > /some/
> > > mailbox timeout. For other cases...
> > >
> > > I'm not too familiar with what other similar drivers may or may not do in
> > > situations like this. The current 256 limit is mostly a reflection of
> > > that being
> > > too small to even support advertised mandatory commands. So things can't
> > > work in
> > > that scenario, but things can work if they have a larger register size
> > > (so long
> > > as the BAR advertises enough space).
> > >
> >
> > I don't think things can work above 1MB, either, though. Section
> > 8.2.8.4.5 specifies 20 bits to define the payload length, if this is
> > larger than cxlm->mbox.payload_size it would venture into the reserved
> > bits of the command register.
> >
> > So is the idea to allow cxl_mem_setup_mailbox() to succeed with a payload
> > size > 1MB and then only check 20 bits for the command register?
>
> So it's probably a spec bug, but actually the payload size is 21 bits... I'll
> check if that was a mistake.
Well I guess they wanted to be able to specify 1M exactly... Spec should
probably say you can't go over 1M
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 03/14] cxl/mem: Find device capabilities
On 21-02-01 14:23:47, David Rientjes wrote:
> On Mon, 1 Feb 2021, Ben Widawsky wrote:
>
> > > > > > +static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm)
> > > > > > +{
> > > > > > + const int cap = cxl_read_mbox_reg32(cxlm,
> > > > > > CXLDEV_MB_CAPS_OFFSET);
> > > > > > +
> > > > > > + cxlm->mbox.payload_size =
> > > > > > + 1 << CXL_GET_FIELD(cap, CXLDEV_MB_CAP_PAYLOAD_SIZE);
> > > > > > +
> > > > > > + /* 8.2.8.4.3 */
> > > > > > + if (cxlm->mbox.payload_size < 256) {
> > > > > > + dev_err(&cxlm->pdev->dev, "Mailbox is too small (%zub)",
> > > > > > + cxlm->mbox.payload_size);
> > > > > > + return -ENXIO;
> > > > > > + }
> > > > >
> > > > > Any reason not to check cxlm->mbox.payload_size > (1 << 20) as well
> > > > > and
> > > > > return ENXIO if true?
> > > >
> > > > If some crazy vendor wanted to ship a mailbox larger than 1M, why
> > > > should the
> > > > driver not allow it?
> > > >
> > >
> > > Because the spec disallows it :)
> >
> > I don't see it being the driver's responsibility to enforce spec correctness
> > though. In certain cases, I need to use the spec, like I have to pick /some/
> > mailbox timeout. For other cases...
> >
> > I'm not too familiar with what other similar drivers may or may not do in
> > situations like this. The current 256 limit is mostly a reflection of that
> > being
> > too small to even support advertised mandatory commands. So things can't
> > work in
> > that scenario, but things can work if they have a larger register size (so
> > long
> > as the BAR advertises enough space).
> >
>
> I don't think things can work above 1MB, either, though. Section
> 8.2.8.4.5 specifies 20 bits to define the payload length, if this is
> larger than cxlm->mbox.payload_size it would venture into the reserved
> bits of the command register.
>
> So is the idea to allow cxl_mem_setup_mailbox() to succeed with a payload
> size > 1MB and then only check 20 bits for the command register?
So it's probably a spec bug, but actually the payload size is 21 bits... I'll
check if that was a mistake.
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 03/14] cxl/mem: Find device capabilities
On 21-02-01 13:51:16, David Rientjes wrote:
> On Mon, 1 Feb 2021, Ben Widawsky wrote:
>
> > On 21-01-30 15:51:49, David Rientjes wrote:
> > > On Fri, 29 Jan 2021, Ben Widawsky wrote:
> > >
> > > > +static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm)
> > > > +{
> > > > + const int cap = cxl_read_mbox_reg32(cxlm,
> > > > CXLDEV_MB_CAPS_OFFSET);
> > > > +
> > > > + cxlm->mbox.payload_size =
> > > > + 1 << CXL_GET_FIELD(cap, CXLDEV_MB_CAP_PAYLOAD_SIZE);
> > > > +
> > > > + /* 8.2.8.4.3 */
> > > > + if (cxlm->mbox.payload_size < 256) {
> > > > + dev_err(&cxlm->pdev->dev, "Mailbox is too small (%zub)",
> > > > + cxlm->mbox.payload_size);
> > > > + return -ENXIO;
> > > > + }
> > >
> > > Any reason not to check cxlm->mbox.payload_size > (1 << 20) as well and
> > > return ENXIO if true?
> >
> > If some crazy vendor wanted to ship a mailbox larger than 1M, why should the
> > driver not allow it?
> >
>
> Because the spec disallows it :)
I don't see it being the driver's responsibility to enforce spec correctness
though. In certain cases, I need to use the spec, like I have to pick /some/
mailbox timeout. For other cases...
I'm not too familiar with what other similar drivers may or may not do in
situations like this. The current 256 limit is mostly a reflection of that being
too small to even support advertised mandatory commands. So things can't work in
that scenario, but things can work if they have a larger register size (so long
as the BAR advertises enough space).
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 09/14] cxl/mem: Add a "RAW" send command
On 21-02-01 13:24:00, Konrad Rzeszutek Wilk wrote: > On Fri, Jan 29, 2021 at 04:24:33PM -0800, Ben Widawsky wrote: > > The CXL memory device send interface will have a number of supported > > commands. The raw command is not such a command. Raw commands allow > > userspace to send a specified opcode to the underlying hardware and > > bypass all driver checks on the command. This is useful for a couple of > > usecases, mainly: > > 1. Undocumented vendor specific hardware commands > > 2. Prototyping new hardware commands not yet supported by the driver > > This sounds like a recipe for .. > > In case you really really want this may I recommend you do two things: > > - Wrap this whole thing with #ifdef > CONFIG_CXL_DEBUG_THIS_WILL_DESTROY_YOUR_LIFE > > (or something equivalant to make it clear this should never be > enabled in production kernels). > > - Add a nice big fat printk in dmesg telling the user that they >are creating a unstable parallel universe that will lead to their >blood pressure going sky-high, or perhaps something more professional >sounding. > > - Rethink this. Do you really really want to encourage vendors > to use this raw API instead of them using the proper APIs? Again, the ideal is proper APIs. Barring that they get a WARN, and a taint if they use the raw commands. > > > > > While this all sounds very powerful it comes with a couple of caveats: > > 1. Bug reports using raw commands will not get the same level of > >attention as bug reports using supported commands (via taint). > > 2. Supported commands will be rejected by the RAW command. > > > > With this comes new debugfs knob to allow full access to your toes with > > your weapon of choice. > > Problem is that debugfs is no longer "debug" but is enabled in > production kernel. I don't see this as my problem. Again, they've been WARNed and tainted. If they want to do this, that's their business. They will be asked to reproduce without RAW if they file a bug report. ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
On 21-02-01 12:54:00, Konrad Rzeszutek Wilk wrote:
> > +#define cxl_doorbell_busy(cxlm)
> > \
> > + (cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET) &\
> > +CXLDEV_MB_CTRL_DOORBELL)
> > +
> > +#define CXL_MAILBOX_TIMEOUT_US 2000
>
> You been using the spec for the values. Is that number also from it ?
>
Yes it is. I'll add a comment with the spec reference.
> > +
> > +enum opcode {
> > + CXL_MBOX_OP_IDENTIFY= 0x4000,
> > + CXL_MBOX_OP_MAX = 0x1
> > +};
> > +
> > +/**
> > + * struct mbox_cmd - A command to be submitted to hardware.
> > + * @opcode: (input) The command set and command submitted to hardware.
> > + * @payload_in: (input) Pointer to the input payload.
> > + * @payload_out: (output) Pointer to the output payload. Must be allocated
> > by
> > + * the caller.
> > + * @size_in: (input) Number of bytes to load from @payload.
> > + * @size_out: (output) Number of bytes loaded into @payload.
> > + * @return_code: (output) Error code returned from hardware.
> > + *
> > + * This is the primary mechanism used to send commands to the hardware.
> > + * All the fields except @payload_* correspond exactly to the fields
> > described in
> > + * Command Register section of the CXL 2.0 spec (8.2.8.4.5). @payload_in
> > and
> > + * @payload_out are written to, and read from the Command Payload Registers
> > + * defined in (8.2.8.4.8).
> > + */
> > +struct mbox_cmd {
> > + u16 opcode;
> > + void *payload_in;
> > + void *payload_out;
>
> On a 32-bit OS (not that we use those that more, but lets assume
> someone really wants to), the void is 4-bytes, while on 64-bit it is
> 8-bytes.
>
> `pahole` is your friend as I think there is a gap between opcode and
> payload_in in the structure.
>
> > + size_t size_in;
> > + size_t size_out;
>
> And those can also change depending on 32-bit/64-bit.
>
> > + u16 return_code;
> > +#define CXL_MBOX_SUCCESS 0
> > +};
>
> Do you want to use __packed to match with the spec?
>
> Ah, reading later you don't care about it.
>
> In that case may I recommend you move 'return_code' (or perhaps just
> call it rc?) to be right after opcode? Less of gaps in that structure.
>
I guess I hadn't realized we're supposed to try to fully pack structs by
default.
> > +
> > +static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> > +{
> > + const int timeout = msecs_to_jiffies(CXL_MAILBOX_TIMEOUT_US);
> > + const unsigned long start = jiffies;
> > + unsigned long end = start;
> > +
> > + while (cxl_doorbell_busy(cxlm)) {
> > + end = jiffies;
> > +
> > + if (time_after(end, start + timeout)) {
> > + /* Check again in case preempted before timeout test */
> > + if (!cxl_doorbell_busy(cxlm))
> > + break;
> > + return -ETIMEDOUT;
> > + }
> > + cpu_relax();
> > + }
>
> Hm, that is not very scheduler friendly. I mean we are sitting here for
> 2000us (2 ms) - that is quite the amount of time spinning.
>
> Should this perhaps be put in a workqueue?
So let me first point you to the friendlier version which was shot down:
https://lore.kernel.org/linux-cxl/2020054356.793390-8-ben.widaw...@intel.com/
I'm not opposed to this being moved to a workqueue at some point, but I think
that's unnecessary complexity currently. The reality is that it's expected that
commands will finish way sooner than this or be implemented as background
commands. I've heard a person who makes a lot of the spec decisions say, "if
it's 2 seconds, nobody will use these things".
I think adding the summary of this back and forth as a comment to the existing
code makes a lot of sense.
> > +
> > + dev_dbg(&cxlm->pdev->dev, "Doorbell wait took %dms",
> > + jiffies_to_msecs(end) - jiffies_to_msecs(start));
> > + return 0;
> > +}
> > +
> > +static void cxl_mem_mbox_timeout(struct cxl_mem *cxlm,
> > +struct mbox_cmd *mbox_cmd)
> > +{
> > + dev_warn(&cxlm->pdev->dev, "Mailbox command timed out\n");
> > + dev_info(&cxlm->pdev->dev,
> > +"\topcode: 0x%04x\n"
> > +"\tpayload size: %zub\n",
> > +mbox_cmd->opcode, mbox_cmd->size_in);
> > +
> > + if (IS_ENABLED(CONFIG_CXL_MEM_INSECURE_DEBUG)) {
> > + print_hex_dump_debug("Payload ", DUMP_PREFIX_OFFSET, 16, 1,
> > +mbox_cmd->payload_in, mbox_cmd->size_in,
> > +true);
> > + }
> > +
> > + /* Here's a good place to figure out if a device reset is needed */
> > +}
> > +
> > +/**
> > + * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory device.
> > + * @cxlm: The CXL memory device to communicate with.
> > + * @mbox_cmd: Command to send to the memory device.
> > + *
> > + * Context: Any context. Expects mbox_lock to be held.
> > + * Return: -ETIMEDOUT if
Re: [PATCH 08/14] taint: add taint for direct hardware access
On 21-02-01 13:18:45, Konrad Rzeszutek Wilk wrote: > On Fri, Jan 29, 2021 at 04:24:32PM -0800, Ben Widawsky wrote: > > For drivers that moderate access to the underlying hardware it is > > sometimes desirable to allow userspace to bypass restrictions. Once > > userspace has done this, the driver can no longer guarantee the sanctity > > of either the OS or the hardware. When in this state, it is helpful for > > kernel developers to be made aware (via this taint flag) of this fact > > for subsequent bug reports. > > > > Example usage: > > - Hardware xyzzy accepts 2 commands, waldo and fred. > > - The xyzzy driver provides an interface for using waldo, but not fred. > > - quux is convinced they really need the fred command. > > - xyzzy driver allows quux to frob hardware to initiate fred. > > Would it not be easier to _not_ frob the hardware for fred-operation? > Aka not implement it or just disallow in the first place? Yeah. So the idea is you either are in a transient phase of the command and some future kernel will have real support for fred - or a vendor is being short sighted and not adding support for fred. > > > > - kernel gets tainted. > > - turns out fred command is borked, and scribbles over memory. > > - developers laugh while closing quux's subsequent bug report. > > Yeah good luck with that theory in-the-field. The customer won't > care about this and will demand a solution for doing fred-operation. > > Just easier to not do fred-operation in the first place,no? The short answer is, in an ideal world you are correct. See nvdimm as an example of the real world. The longer answer. Unless we want to wait until we have all the hardware we're ever going to see, it's impossible to have a fully baked, and validated interface. The RAW interface is my admission that I make no guarantees about being able to provide the perfect interface and giving the power back to the hardware vendors and their driver writers. As an example, suppose a vendor shipped a device with their special vendor opcode. They can enable their customers to use that opcode on any driver version. That seems pretty powerful and worthwhile to me. Or a more realistic example, we ship a driver that adds a command which is totally broken. Customers can utilize the RAW interface until it gets fixed in a subsequent release which might be quite a ways out. I'll say the RAW interface isn't an encouraged usage, but it's one that I expect to be needed, and if it's not we can always try to kill it later. If nobody is actually using it, nobody will complain, right :D ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 03/14] cxl/mem: Find device capabilities
On 21-02-01 12:41:36, Konrad Rzeszutek Wilk wrote:
> > +static int cxl_mem_setup_regs(struct cxl_mem *cxlm)
> > +{
> > + struct device *dev = &cxlm->pdev->dev;
> > + int cap, cap_count;
> > + u64 cap_array;
> > +
> > + cap_array = readq(cxlm->regs + CXLDEV_CAP_ARRAY_OFFSET);
> > + if (CXL_GET_FIELD(cap_array, CXLDEV_CAP_ARRAY_ID) !=
> > CXLDEV_CAP_ARRAY_CAP_ID)
> > + return -ENODEV;
> > +
> > + cap_count = CXL_GET_FIELD(cap_array, CXLDEV_CAP_ARRAY_COUNT);
> > +
> > + for (cap = 1; cap <= cap_count; cap++) {
> > + void __iomem *register_block;
> > + u32 offset;
> > + u16 cap_id;
> > +
> > + cap_id = readl(cxlm->regs + cap * 0x10) & 0x;
> > + offset = readl(cxlm->regs + cap * 0x10 + 0x4);
> > + register_block = cxlm->regs + offset;
> > +
> > + switch (cap_id) {
> > + case CXLDEV_CAP_CAP_ID_DEVICE_STATUS:
> > + dev_dbg(dev, "found Status capability (0x%x)\n",
> > + offset);
>
> That 80 character limit is no longer a requirement. Can you just make
> this one line? And perhaps change 'found' to 'Found' ?
>
Funny that.
https://lore.kernel.org/linux-cxl/2020073449.ga16...@infradead.org/
> > + cxlm->status.regs = register_block;
> > + break;
> > + case CXLDEV_CAP_CAP_ID_PRIMARY_MAILBOX:
> > + dev_dbg(dev, "found Mailbox capability (0x%x)\n",
> > + offset);
> > + cxlm->mbox.regs = register_block;
> > + break;
> > + case CXLDEV_CAP_CAP_ID_SECONDARY_MAILBOX:
> > + dev_dbg(dev,
> > + "found Secondary Mailbox capability (0x%x)\n",
> > + offset);
> > + break;
> > + case CXLDEV_CAP_CAP_ID_MEMDEV:
> > + dev_dbg(dev, "found Memory Device capability (0x%x)\n",
> > + offset);
> > + cxlm->mem.regs = register_block;
> > + break;
> > + default:
> > + dev_warn(dev, "Unknown cap ID: %d (0x%x)\n", cap_id,
> > +offset);
> > + break;
> > + }
> > + }
> > +
> > + if (!cxlm->status.regs || !cxlm->mbox.regs || !cxlm->mem.regs) {
> > + dev_err(dev, "registers not found: %s%s%s\n",
> > + !cxlm->status.regs ? "status " : "",
> > + !cxlm->mbox.regs ? "mbox " : "",
> > + !cxlm->mem.regs ? "mem" : "");
> > + return -ENXIO;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm)
> > +{
> > + const int cap = cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CAPS_OFFSET);
> > +
> > + cxlm->mbox.payload_size =
> > + 1 << CXL_GET_FIELD(cap, CXLDEV_MB_CAP_PAYLOAD_SIZE);
> > +
>
> I think the static analyzers are not going to be happy that you are not
> checking the value of `cap` before using it.
>
> Perhaps you should check that first before doing the manipulations?
>
I'm not following the request. CXL_GET_FIELD is just doing the shift and mask on
cap.
Can you explain what you're hoping to see?
> > + /* 8.2.8.4.3 */
> > + if (cxlm->mbox.payload_size < 256) {
>
> #define for 256?
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 05/14] cxl/mem: Register CXL memX devices
On 21-01-30 15:52:01, David Rientjes wrote: > On Fri, 29 Jan 2021, Ben Widawsky wrote: > > > diff --git a/Documentation/ABI/testing/sysfs-bus-cxl > > b/Documentation/ABI/testing/sysfs-bus-cxl > > new file mode 100644 > > index ..fe7b87eba988 > > --- /dev/null > > +++ b/Documentation/ABI/testing/sysfs-bus-cxl > > @@ -0,0 +1,26 @@ > > +What: /sys/bus/cxl/devices/memX/firmware_version > > +Date: December, 2020 > > +KernelVersion: v5.12 > > +Contact: linux-...@vger.kernel.org > > +Description: > > + (RO) "FW Revision" string as reported by the Identify > > + Memory Device Output Payload in the CXL-2.0 > > + specification. > > + > > +What: /sys/bus/cxl/devices/memX/ram/size > > +Date: December, 2020 > > +KernelVersion: v5.12 > > +Contact: linux-...@vger.kernel.org > > +Description: > > + (RO) "Volatile Only Capacity" as reported by the > > + Identify Memory Device Output Payload in the CXL-2.0 > > + specification. > > + > > +What: /sys/bus/cxl/devices/memX/pmem/size > > +Date: December, 2020 > > +KernelVersion: v5.12 > > +Contact: linux-...@vger.kernel.org > > +Description: > > + (RO) "Persistent Only Capacity" as reported by the > > + Identify Memory Device Output Payload in the CXL-2.0 > > + specification. > > Aren't volatile and persistent capacities expressed in multiples of 256MB? As of the spec today, volatile and persistent capacities are required to be in multiples of 256MB, however, future specs may not have such a requirement and I think keeping sysfs ABI easily forward portable makes sense. ___ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
Re: [PATCH 03/14] cxl/mem: Find device capabilities
On 21-01-30 15:51:49, David Rientjes wrote:
> On Fri, 29 Jan 2021, Ben Widawsky wrote:
>
> > +static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm)
> > +{
> > + const int cap = cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CAPS_OFFSET);
> > +
> > + cxlm->mbox.payload_size =
> > + 1 << CXL_GET_FIELD(cap, CXLDEV_MB_CAP_PAYLOAD_SIZE);
> > +
> > + /* 8.2.8.4.3 */
> > + if (cxlm->mbox.payload_size < 256) {
> > + dev_err(&cxlm->pdev->dev, "Mailbox is too small (%zub)",
> > + cxlm->mbox.payload_size);
> > + return -ENXIO;
> > + }
>
> Any reason not to check cxlm->mbox.payload_size > (1 << 20) as well and
> return ENXIO if true?
If some crazy vendor wanted to ship a mailbox larger than 1M, why should the
driver not allow it?
I'm open to changing it, it just seems like a larger mailbox wouldn't be fatal.
___
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-le...@lists.01.org
