This series aims to address long-standing conflicts between HMEM and CXL when handling Soft Reserved memory ranges.
Reworked from Dan's patch: https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/patch/?id=ab70c6227ee6165a562c215d9dcb4a1c55620d5d Previous work: https://lore.kernel.org/all/[email protected]/ Link to v5: https://lore.kernel.org/all/[email protected] The series is based on branch "for-7.0/cxl-init" and base-commit is base-commit: bc62f5b308cbdedf29132fe96e9d591e526527e1 [1] After offlining the memory I can tear down the regions and recreate them back. dax_cxl creates dax devices and onlines memory. 850000000-284fffffff : CXL Window 0 850000000-284fffffff : region0 850000000-284fffffff : dax0.0 850000000-284fffffff : System RAM (kmem) [2] With CONFIG_CXL_REGION disabled, all the resources are handled by HMEM. Soft Reserved range shows up in /proc/iomem, no regions come up and dax devices are created from HMEM. 850000000-284fffffff : CXL Window 0 850000000-284fffffff : Soft Reserved 850000000-284fffffff : dax0.0 850000000-284fffffff : System RAM (kmem) [3] Region assembly failure works same as [2]. [4] REGISTER path: When CXL_BUS = y (with CXL_ACPI, CXL_PCI, CXL_PORT, CXL_MEM = y), the dax_cxl driver is probed and completes initialization before dax_hmem probes. This scenario was tested with CXL = y, DAX_CXL = m and DAX_HMEM = m. To validate the REGISTER path, I forced REGISTER even in cases where SR completely overlaps the CXL region as I did not have access to a system where the CXL region range is smaller than the SR range. 850000000-284fffffff : Soft Reserved 850000000-284fffffff : CXL Window 0 850000000-280fffffff : region0 850000000-284fffffff : dax0.0 850000000-284fffffff : System RAM (kmem) "path":"\/platform\/ACPI0017:00\/root0\/decoder0.0\/region0\/dax_region0", "id":0, "size":"128.00 GiB (137.44 GB)", "align":2097152 [ 35.961707] cxl-dax: cxl_dax_region_init() [ 35.961713] cxl-dax: registering driver. [ 35.961715] cxl-dax: dax_hmem work flushed. [ 35.961754] alloc_dev_dax_range: dax0.0: alloc range[0]: 0x000000850000000:0x000000284fffffff [ 35.976622] hmem: hmem_platform probe started. [ 35.980821] cxl_bus_probe: cxl_dax_region dax_region0: probe: 0 [ 36.819566] hmem_platform hmem_platform.0: Soft Reserved not fully contained in CXL; using HMEM [ 36.819569] hmem_register_device: hmem_platform hmem_platform.0: registering CXL range: [mem 0x850000000-0x284fffffff flags 0x80000200] [ 36.934156] alloc_dax_region: hmem hmem.6: dax_region resource conflict for [mem 0x850000000-0x284fffffff] [ 36.989310] hmem hmem.6: probe with driver hmem failed with error -12 [5] When CXL_BUS = m (with CXL_ACPI, CXL_PCI, CXL_PORT, CXL_MEM = m), DAX_CXL = m and DAX_HMEM = y the results are as expected. To validate the REGISTER path, I forced REGISTER even in cases where SR completely overlaps the CXL region as I did not have access to a system where the CXL region range is smaller than the SR range. 850000000-284fffffff : Soft Reserved 850000000-284fffffff : CXL Window 0 850000000-280fffffff : region0 850000000-284fffffff : dax6.0 850000000-284fffffff : System RAM (kmem) "path":"\/platform\/hmem.6", "id":6, "size":"128.00 GiB (137.44 GB)", "align":2097152 [ 30.897665] devm_cxl_add_dax_region: cxl_region region0: region0: register dax_region0 [ 30.921015] hmem: hmem_platform probe started. [ 31.017946] hmem_platform hmem_platform.0: Soft Reserved not fully contained in CXL; using HMEM [ 31.056310] alloc_dev_dax_range: dax6.0: alloc range[0]: 0x0000000850000000:0x000000284fffffff [ 34.781516] cxl-dax: cxl_dax_region_init() [ 34.781522] cxl-dax: registering driver. [ 34.781523] cxl-dax: dax_hmem work flushed. [ 34.781549] alloc_dax_region: cxl_dax_region dax_region0: dax_region resource conflict for [mem 0x850000000-0x284fffffff] [ 34.781552] cxl_bus_probe: cxl_dax_region dax_region0: probe: -12 [ 34.781554] cxl_dax_region dax_region0: probe with driver cxl_dax_region failed with error -12 v6 updates: - Patch 1-3 no changes. - New Patches 4-5. - (void *)res -> res. - cxl_region_contains_soft_reserve -> region_contains_soft_reserve. - New file include/cxl/cxl.h - Introduced singleton workqueue. - hmem to queue the work and cxl to flush. - cxl_contains_soft_reserve() -> soft_reserve_has_cxl_match(). - Included descriptions for dax_cxl_mode. - kzalloc -> kmalloc in add_soft_reserve_into_iomem() - dax_cxl_mode is exported to CXL. - Introduced hmem_register_cxl_device() for walking only CXL intersected SR ranges the second time. v5 updates: - Patch 1 dropped as its been merged for-7.0/cxl-init. - Added Reviewed-by tags. - Shared dax_cxl_mode between dax/cxl.c and dax/hmem.c and used -EPROBE_DEFER to defer dax_cxl. - CXL_REGION_F_AUTO check for resetting decoders. - Teardown all CXL regions if any one CXL region doesn't fully contain the Soft Reserved range. - Added helper cxl_region_contains_sr() to determine Soft Reserved ownership. - bus_rescan_devices() to retry dax_cxl. - Added guard(rwsem_read)(&cxl_rwsem.region). v4 updates: - No changes patches 1-3. - New patches 4-7. - handle_deferred_cxl() has been enhanced to handle case where CXL regions do not contiguously and fully cover Soft Reserved ranges. - Support added to defer cxl_dax registration. - Support added to teardown cxl regions. v3 updates: - Fixed two "From". v2 updates: - Removed conditional check on CONFIG_EFI_SOFT_RESERVE as dax_hmem depends on CONFIG_EFI_SOFT_RESERVE. (Zhijian) - Added TODO note. (Zhijian) - Included region_intersects_soft_reserve() inside CONFIG_EFI_SOFT_RESERVE conditional check. (Zhijian) - insert_resource_late() -> insert_resource_expand_to_fit() and __insert_resource_expand_to_fit() replacement. (Boris) - Fixed Co-developed and Signed-off by. (Dan) - Combined 2/6 and 3/6 into a single patch. (Zhijian). - Skip local variable in remove_soft_reserved. (Jonathan) - Drop kfree with __free(). (Jonathan) - return 0 -> return dev_add_action_or_reset(host...) (Jonathan) - Dropped 6/6. - Reviewed-by tags (Dave, Jonathan) Dan Williams (3): dax/hmem: Request cxl_acpi and cxl_pci before walking Soft Reserved ranges dax/hmem: Gate Soft Reserved deferral on DEV_DAX_CXL dax/cxl, hmem: Initialize hmem early and defer dax_cxl binding Smita Koralahalli (6): cxl/region: Skip decoder reset on detach for autodiscovered regions dax: Track all dax_region allocations under a global resource tree cxl/region: Add helper to check Soft Reserved containment by CXL regions dax: Add deferred-work helpers for dax_hmem and dax_cxl coordination dax/hmem, cxl: Defer and resolve ownership of Soft Reserved memory ranges dax/hmem: Reintroduce Soft Reserved ranges back into the iomem tree drivers/cxl/core/region.c | 34 +++++++++- drivers/dax/Kconfig | 2 + drivers/dax/Makefile | 3 +- drivers/dax/bus.c | 84 ++++++++++++++++++++++++- drivers/dax/bus.h | 26 ++++++++ drivers/dax/cxl.c | 28 ++++++++- drivers/dax/hmem/hmem.c | 129 ++++++++++++++++++++++++++++++++++---- include/cxl/cxl.h | 15 +++++ 8 files changed, 303 insertions(+), 18 deletions(-) create mode 100644 include/cxl/cxl.h -- 2.17.1
