Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Tue, Jul 25, 2023 at 08:18:08AM -0700, Ira Weiny wrote: > Fan Ni wrote: > > On Thu, May 11, 2023 at 05:56:40PM +, Fan Ni wrote: > > > > FYI. > > > > I have updated the patch series and sent out again. > > > > I suggested anyone who are interested in DCD and using this patch series to > > use the new series. Quite a few things has been fixed. > > > > https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230724162313.34196-1-fan...@samsung.com/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!V8kDTpT5yLUAyWm3sFm7XIgN0QdNUyQYZd9vYLHjUVkMkhDT14F8avgNBh23KPAtsS_dGm2LZuHJ102mgIg$ > > > > > > Also, if you want to use the code repo directly, you can try > > > > https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=0600fd6b-678be820-06017624-74fe485fb305-f529279062b02b73=1=5df65010-e62f-40a1-9a21-609eb1400921=https*3A*2F*2Fgithub.com*2Fmoking*2Fqemu-dcd-preview-latest*2Ftree*2Fdcd-dev__;JSUlJSUlJQ!!EwVzqGoTKBqv-0DWAJBm!V8kDTpT5yLUAyWm3sFm7XIgN0QdNUyQYZd9vYLHjUVkMkhDT14F8avgNBh23KPAtsS_dGm2LZuHJpB0SmNs$ > > > > Thanks for the branch! > > I took a quick look and I don't see a resolution to the problem I > mentioned with non DCD devices being supported.[1] > > [1] > https://urldefense.com/v3/__https://lore.kernel.org/all/6483946e8152f_f1132294a2@iweiny-mobl.notmuch/__;!!EwVzqGoTKBqv-0DWAJBm!V8kDTpT5yLUAyWm3sFm7XIgN0QdNUyQYZd9vYLHjUVkMkhDT14F8avgNBh23KPAtsS_dGm2LZuHJ4geSEAg$ > > > Did you fix this in a different way? If I don't add DC to my mem devices they > don't get probed properly. I'm still looking into this with your new branch, > but I don't think DC commands should be in the CEL if the device does not > support it. > > Also I get a build warning on this branch I had to fix[3] as my build is > treating warnings as errors.[2] > > I don't think this fix is technically necessary as 'list' should never be NULL > that I can see. But might be nice to check or just use my fix. > > I'll try and get to a review once I get the DCD stuff out on the list again. > > Ira > Oh, I missed your previous comments, let me look into it and fix accordingly and send out a new version. Btw, when I did the DCD test with the last DCD kernel code, I found some issue there. When I add a DCD extent for the first time, it will be recognized as system RAM automatically and show up with command lsmem. However, when I release it and try to re-add the same extent again. The adding seems normal and the device will show up under /dev/ as dax0.X. But it will not show up with lsmem command and I have to use daxctl reconfigure command to turn it to system ram and then it can show up with lsmem command. I would expect the behavior for the first add and second add be the same. Fan. > > [2] > ../hw/mem/cxl_type3.c: In function > ‘qmp_cxl_process_dynamic_capacity_event.constprop’: > ../hw/mem/cxl_type3.c:2063:28: error: ‘rid’ may be used uninitialized > [-Werror=maybe-uninitialized] > 2063 | dCap.updated_region_id = rid; > | ~~~^ > ../hw/mem/cxl_type3.c:1987:13: note: ‘rid’ was declared here > 1987 | uint8_t rid; > | ^~~ > cc1: all warnings being treated as errors > > [3] > > diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c > index e67328780407..d25e6064f6c9 100644 > --- a/hw/mem/cxl_type3.c > +++ b/hw/mem/cxl_type3.c > @@ -1984,7 +1984,7 @@ static void > qmp_cxl_process_dynamic_capacity_event(const char *path, > CXLDCExtentRecordList *list = records; > CXLDCExtent_raw *extents; > uint64_t dpa, len; > -uint8_t rid; > +uint8_t rid = 0; > int i; > > if (!obj) {
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
Fan Ni wrote: > On Thu, May 11, 2023 at 05:56:40PM +, Fan Ni wrote: > > FYI. > > I have updated the patch series and sent out again. > > I suggested anyone who are interested in DCD and using this patch series to > use the new series. Quite a few things has been fixed. > > https://lore.kernel.org/linux-cxl/20230724162313.34196-1-fan...@samsung.com/T/#t > > Also, if you want to use the code repo directly, you can try > > https://github.com/moking/qemu-dcd-preview-latest/tree/dcd-dev Thanks for the branch! I took a quick look and I don't see a resolution to the problem I mentioned with non DCD devices being supported.[1] [1] https://lore.kernel.org/all/6483946e8152f_f1132294a2@iweiny-mobl.notmuch/ Did you fix this in a different way? If I don't add DC to my mem devices they don't get probed properly. I'm still looking into this with your new branch, but I don't think DC commands should be in the CEL if the device does not support it. Also I get a build warning on this branch I had to fix[3] as my build is treating warnings as errors.[2] I don't think this fix is technically necessary as 'list' should never be NULL that I can see. But might be nice to check or just use my fix. I'll try and get to a review once I get the DCD stuff out on the list again. Ira [2] ../hw/mem/cxl_type3.c: In function ‘qmp_cxl_process_dynamic_capacity_event.constprop’: ../hw/mem/cxl_type3.c:2063:28: error: ‘rid’ may be used uninitialized [-Werror=maybe-uninitialized] 2063 | dCap.updated_region_id = rid; | ~~~^ ../hw/mem/cxl_type3.c:1987:13: note: ‘rid’ was declared here 1987 | uint8_t rid; | ^~~ cc1: all warnings being treated as errors [3] diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index e67328780407..d25e6064f6c9 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -1984,7 +1984,7 @@ static void qmp_cxl_process_dynamic_capacity_event(const char *path, CXLDCExtentRecordList *list = records; CXLDCExtent_raw *extents; uint64_t dpa, len; -uint8_t rid; +uint8_t rid = 0; int i; if (!obj) {
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Thu, May 11, 2023 at 05:56:40PM +, Fan Ni wrote: FYI. I have updated the patch series and sent out again. I suggested anyone who are interested in DCD and using this patch series to use the new series. Quite a few things has been fixed. https://lore.kernel.org/linux-cxl/20230724162313.34196-1-fan...@samsung.com/T/#t Also, if you want to use the code repo directly, you can try https://github.com/moking/qemu-dcd-preview-latest/tree/dcd-dev Fan > Since the early draft of DCD support in kernel is out > (https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/#t), > this patch series provide dcd emulation in qemu so people who are interested > can have an early try. It is noted that the patch series may need to be > updated > accordingly if the kernel side implementation changes. > > To support DCD emulation, the patch series add DCD related mailbox command > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device > with dynamic capacity extent and region representative. > To support read/write to the dynamic capacity of the device, a host backend > is provided and necessary check mechnism is added to ensure the dynamic > capacity accessed is backed with active dc extents. > Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not supported > , but we add two qmp interfaces for adding/releasing dynamic capacity extents. > Also, the support for multiple hosts sharing the same DCD case is missing. > > Things we can try with the patch series together with kernel dcd code: > 1. Create DC regions to cover the address range of the dynamic capacity > regions. > 2. Add/release dynamic capacity extents to the device and notify the > kernel. > 3. Test kernel side code to accept added dc extents and create dax devices, > and release dc extents and notify the device > 4. Online the memory range backed with dc extents and let application use > them. > > The patch series is based on Jonathan's local qemu branch: > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28 > > Simple tests peformed with the patch series: > 1 Install cxl modules: > > modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem > > 2 Create dc regions: > > region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) > echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region > echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity > echo 1 > /sys/bus/cxl/devices/$region/interleave_ways > echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode > echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size > echo 0x1000 > /sys/bus/cxl/devices/$region/size > echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 > echo 1 > /sys/bus/cxl/devices/$region/commit > echo $region > /sys/bus/cxl/drivers/cxl_region/bind > > /home/fan/cxl/tools-and-scripts# cxl list > [ > { > "memdevs":[ > { > "memdev":"mem0", > "pmem_size":536870912, > "ram_size":0, > "serial":0, > "host":":0d:00.0" > } > ] > }, > { > "regions":[ > { > "region":"region0", > "resource":45365592064, > "size":268435456, > "interleave_ways":1, > "interleave_granularity":256, > "decode_state":"commit" > } > ] > } > ] > > 3 Add two dc extents (128MB each) through qmp interface > > { "execute": "qmp_capabilities" } > > { "execute": "cxl-add-dynamic-capacity-event", > "arguments": { >"path": "/machine/peripheral/cxl-pmem0", > "region-id" : 0, >"num-extent": 2, > "dpa":0, > "extent-len": 128 > } > } > > /home/fan/cxl/tools-and-scripts# lsmem > RANGE SIZE STATE REMOVABLE BLOCK > 0x-0x7fff2G online yes0-15 > 0x0001-0x00027fff6G online yes 32-79 > 0x000a9000-0x000a9fff 256M offline 338-339 > > Memory block size: 128M > Total online memory: 8G > Total offline memory:256M > > > 4.Online the momory with 'daxctl online-memory dax0.0' to online the memory > > /home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0 > [ 230.730553] Fallback order for Node 0: 0 1 > [ 230.730825] Fallback order for Node 1: 1 0 > [ 230.730953] Built 2 zonelists, mobility grouping on. Total pages: 2042541 > [ 230.731110] Policy zone: Normal > onlined memory for 1 device > > root@bgt-140510-bm03:/home/fan/cxl/ndctl# lsmem > RANGE SIZE STATE REMOVABLE BLOCK > 0x-0x7fff2G online yes 0-15 > 0x0001-0x00027fff6G online yes 32-79 > 0x000a9000-0x000a97ff 128M online yes 338 > 0x000a9800-0x000a9fff 128M offline 339 > > Memory block size: 128M > Total online memory: 8.1G > Total offline memory:
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
The 05/15/2023 14:00, Jonathan Cameron wrote: > On Thu, 11 May 2023 17:56:40 + > Fan Ni wrote: > > > Since the early draft of DCD support in kernel is out > > (https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/#t), > > this patch series provide dcd emulation in qemu so people who are interested > > can have an early try. It is noted that the patch series may need to be > > updated > > accordingly if the kernel side implementation changes. > > > > To support DCD emulation, the patch series add DCD related mailbox command > > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device > > with dynamic capacity extent and region representative. > > To support read/write to the dynamic capacity of the device, a host backend > > is provided and necessary check mechnism is added to ensure the dynamic > > capacity accessed is backed with active dc extents. > > Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not > > supported > > , but we add two qmp interfaces for adding/releasing dynamic capacity > > extents. > > Also, the support for multiple hosts sharing the same DCD case is missing. > > > > Things we can try with the patch series together with kernel dcd code: > > 1. Create DC regions to cover the address range of the dynamic capacity > > regions. > > 2. Add/release dynamic capacity extents to the device and notify the > > kernel. > > 3. Test kernel side code to accept added dc extents and create dax devices, > > and release dc extents and notify the device > > 4. Online the memory range backed with dc extents and let application use > > them. > > > > The patch series is based on Jonathan's local qemu branch: > > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28 > > > > Simple tests peformed with the patch series: > > 1 Install cxl modules: > > > > modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem > > > > 2 Create dc regions: > > > > region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) > > echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region > > echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity > > echo 1 > /sys/bus/cxl/devices/$region/interleave_ways > > echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode > > echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size > > echo 0x1000 > /sys/bus/cxl/devices/$region/size > > echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 > > echo 1 > /sys/bus/cxl/devices/$region/commit > > echo $region > /sys/bus/cxl/drivers/cxl_region/bind > > > > /home/fan/cxl/tools-and-scripts# cxl list > > [ > > { > > "memdevs":[ > > { > > "memdev":"mem0", > > "pmem_size":536870912, > > "ram_size":0, > > "serial":0, > > "host":":0d:00.0" > > } > > ] > > }, > > { > > "regions":[ > > { > > "region":"region0", > > "resource":45365592064, > > "size":268435456, > > "interleave_ways":1, > > "interleave_granularity":256, > > "decode_state":"commit" > > } > > ] > > } > > ] > > > > 3 Add two dc extents (128MB each) through qmp interface > > > > { "execute": "qmp_capabilities" } > > > > { "execute": "cxl-add-dynamic-capacity-event", > > "arguments": { > > "path": "/machine/peripheral/cxl-pmem0", > > "region-id" : 0, > > "num-extent": 2, > > "dpa":0, > > "extent-len": 128 > > } > > } > > > > /home/fan/cxl/tools-and-scripts# lsmem > > RANGE SIZE STATE REMOVABLE BLOCK > > 0x-0x7fff2G online yes0-15 > > 0x0001-0x00027fff6G online yes 32-79 > > 0x000a9000-0x000a9fff 256M offline 338-339 > > > > Memory block size: 128M > > Total online memory: 8G > > Total offline memory:256M > > > > > > 4.Online the momory with 'daxctl online-memory dax0.0' to online the memory > > > > /home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0 > > [ 230.730553] Fallback order for Node 0: 0 1 > > [ 230.730825] Fallback order for Node 1: 1 0 > > [ 230.730953] Built 2 zonelists, mobility grouping on. Total pages: > > 2042541 > > [ 230.731110] Policy zone: Normal > > onlined memory for 1 device > > > > root@bgt-140510-bm03:/home/fan/cxl/ndctl# lsmem > > RANGE SIZE STATE REMOVABLE BLOCK > > 0x-0x7fff2G online yes 0-15 > > 0x0001-0x00027fff6G online yes 32-79 > > 0x000a9000-0x000a97ff 128M online yes 338 > > 0x000a9800-0x000a9fff 128M offline 339 > > > > Memory block size: 128M > > Total online memory: 8.1G > > Total offline memory:128M > > > > 5 using dc extents as regular memory > > > > /home/fan/cxl/ndctl# numactl --membind=1 ls > >
Re: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
From: Ira Weiny Sent: Friday, June 9, 2023 2:06 PM To: ni...@outlook.com ; Ira Weiny ; Shesha Bhushan Sreenivasamurthy Cc: Shesha Bhushan Sreenivasamurthy ; Fan Ni ; Jonathan Cameron ; qemu-devel@nongnu.org ; linux-...@vger.kernel.org ; gregory.pr...@memverge.com ; hch...@avery-design.com.tw ; cbr...@avery-design.com ; dan.j.willi...@intel.com ; Adam Manzanares ; d...@stgolabs.net ; nmtadam.sams...@gmail.com Subject: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu External Email -- ni...@outlook.com wrote: > The 06/08/2023 08:43, Ira Weiny wrote: > > Shesha Bhushan Sreenivasamurthy wrote: [snip] > > Hi Ira & Shesha, > FYI. I reabased my patch series on top of the above branch and created a new > branch here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_moking_qemu-2Ddcd-2Dpreview-2Dlatest_tree_dcd-2Dpreview=DwIBAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=SvyB_49EIFUT8-ZEVgIEYYjU6-zGTX4wb30kuNLUhkTSHYZK5-C0Gxr7uvefhtj4=MFD7qlSaTuy-w6aDmavIMbSP_aeaqZmSML7IVOX5jLs= > Thanks! > > It passes the same tests as shown here: > https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_linux-2Dcxl_6481f70fca5c2-5Fc82be29440-40iweiny-2Dmobl.notmuch_T_-23m76f6e85ce3d7292b1982960eb22086ee03922166=DwIBAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=SvyB_49EIFUT8-ZEVgIEYYjU6-zGTX4wb30kuNLUhkTSHYZK5-C0Gxr7uvefhtj4=e-fQOi0RzSZXxfSz37Bpz1sKtp7Yy0MWqonZnswK0RU= > I've not gotten very far with this testing. But I did find that regular type 3 devices don't work with this change. I used the patch below to get this working. Was there something I was missing to configure a non-DCD device? I don't particularly like adding another bool to this call stack. Seems like this calls for a flags field but I want to move on to DCD work so I hacked this in. I am working on the DCD FM-API commands here - https://gitlab.com/sheshas/qemu-fmapi/-/tree/cxl-2023-05-25 -Shesha Ira commit ed27935044dcbd2c6ba71f8411b218621f3f4167 Author: Ira Weiny Date: Fri Jun 9 13:56:33 2023 -0700 hw/mem/cxl_type3: Exclude DCD from CEL when type3 is not DCD Per CXL 3.0 9.13.3 Dynamic Capacity Device (DCD) when the type 3 memory device does not have DCD support the CEL should not include DCD configuration commands. If the number of DC regions supported is 0 skip the DCD commands in the CEL. Applies on top of Fan Ni's work here: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_moking_qemu-2Ddcd-2Dpreview-2Dlatest_tree_dcd-2Dpreview=DwIBAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=SvyB_49EIFUT8-ZEVgIEYYjU6-zGTX4wb30kuNLUhkTSHYZK5-C0Gxr7uvefhtj4=MFD7qlSaTuy-w6aDmavIMbSP_aeaqZmSML7IVOX5jLs= Not-yet-Signed-off-by: Ira Weiny diff --git a/hw/cxl/cxl-device-utils.c b/hw/cxl/cxl-device-utils.c index a4a2c6a80004..262e35935563 100644 --- a/hw/cxl/cxl-device-utils.c +++ b/hw/cxl/cxl-device-utils.c @@ -288,7 +288,7 @@ static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate) static void memdev_reg_init_common(CXLDeviceState *cxl_dstate) { } -void cxl_device_register_init_common(CXLDeviceState *cxl_dstate) +void cxl_device_register_init_common(CXLDeviceState *cxl_dstate, bool is_dcd) { uint64_t *cap_hdrs = cxl_dstate->caps_reg_state64; const int cap_count = 3; @@ -307,7 +307,7 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate) cxl_device_cap_init(cxl_dstate, MEMORY_DEVICE, 0x4000, 1); memdev_reg_init_common(cxl_dstate); - cxl_initialize_mailbox(cxl_dstate, false); + cxl_initialize_mailbox(cxl_dstate, false, is_dcd); } void cxl_device_register_init_swcci(CXLDeviceState *cxl_dstate) @@ -329,7 +329,7 @@ void cxl_device_register_init_swcci(CXLDeviceState *cxl_dstate) cxl_device_cap_init(cxl_dstate, MEMORY_DEVICE, 0x4000, 1); memdev_reg_init_common(cxl_dstate); - cxl_initialize_mailbox(cxl_dstate, true); + cxl_initialize_mailbox(cxl_dstate, true, false); } uint64_t cxl_device_get_timestamp(CXLDeviceState *cxl_dstate) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 93b26e717c94..80e9cb9a8f04 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1526,7 +1526,8 @@ static void bg_timercb(void *opaque) } } -void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate, bool switch_cci) +void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate, bool switch_cci, + bool is_dcd) { if (!switch_cci) { cxl_dstate->cxl_cmd_set = cxl_cmd_set; @@ -1534,6 +1535,9 @@ void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate, bool switch_cci) cxl_dstate->cxl_cmd_set = cxl_cmd_set_sw; } for (int set = 0; set < 256; set++) { +
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
ni...@outlook.com wrote: > The 06/08/2023 08:43, Ira Weiny wrote: > > Shesha Bhushan Sreenivasamurthy wrote: [snip] > > Hi Ira & Shesha, > FYI. I reabased my patch series on top of the above branch and created a new > branch here: > > https://github.com/moking/qemu-dcd-preview-latest/tree/dcd-preview Thanks! > > It passes the same tests as shown here: > https://lore.kernel.org/linux-cxl/6481f70fca5c2_c82be29440@iweiny-mobl.notmuch/T/#m76f6e85ce3d7292b1982960eb22086ee03922166 I've not gotten very far with this testing. But I did find that regular type 3 devices don't work with this change. I used the patch below to get this working. Was there something I was missing to configure a non-DCD device? I don't particularly like adding another bool to this call stack. Seems like this calls for a flags field but I want to move on to DCD work so I hacked this in. Ira commit ed27935044dcbd2c6ba71f8411b218621f3f4167 Author: Ira Weiny Date: Fri Jun 9 13:56:33 2023 -0700 hw/mem/cxl_type3: Exclude DCD from CEL when type3 is not DCD Per CXL 3.0 9.13.3 Dynamic Capacity Device (DCD) when the type 3 memory device does not have DCD support the CEL should not include DCD configuration commands. If the number of DC regions supported is 0 skip the DCD commands in the CEL. Applies on top of Fan Ni's work here: https://github.com/moking/qemu-dcd-preview-latest/tree/dcd-preview Not-yet-Signed-off-by: Ira Weiny diff --git a/hw/cxl/cxl-device-utils.c b/hw/cxl/cxl-device-utils.c index a4a2c6a80004..262e35935563 100644 --- a/hw/cxl/cxl-device-utils.c +++ b/hw/cxl/cxl-device-utils.c @@ -288,7 +288,7 @@ static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate) static void memdev_reg_init_common(CXLDeviceState *cxl_dstate) { } -void cxl_device_register_init_common(CXLDeviceState *cxl_dstate) +void cxl_device_register_init_common(CXLDeviceState *cxl_dstate, bool is_dcd) { uint64_t *cap_hdrs = cxl_dstate->caps_reg_state64; const int cap_count = 3; @@ -307,7 +307,7 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate) cxl_device_cap_init(cxl_dstate, MEMORY_DEVICE, 0x4000, 1); memdev_reg_init_common(cxl_dstate); -cxl_initialize_mailbox(cxl_dstate, false); +cxl_initialize_mailbox(cxl_dstate, false, is_dcd); } void cxl_device_register_init_swcci(CXLDeviceState *cxl_dstate) @@ -329,7 +329,7 @@ void cxl_device_register_init_swcci(CXLDeviceState *cxl_dstate) cxl_device_cap_init(cxl_dstate, MEMORY_DEVICE, 0x4000, 1); memdev_reg_init_common(cxl_dstate); -cxl_initialize_mailbox(cxl_dstate, true); +cxl_initialize_mailbox(cxl_dstate, true, false); } uint64_t cxl_device_get_timestamp(CXLDeviceState *cxl_dstate) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 93b26e717c94..80e9cb9a8f04 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1526,7 +1526,8 @@ static void bg_timercb(void *opaque) } } -void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate, bool switch_cci) +void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate, bool switch_cci, +bool is_dcd) { if (!switch_cci) { cxl_dstate->cxl_cmd_set = cxl_cmd_set; @@ -1534,6 +1535,9 @@ void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate, bool switch_cci) cxl_dstate->cxl_cmd_set = cxl_cmd_set_sw; } for (int set = 0; set < 256; set++) { +if (!is_dcd && set == DCD_CONFIG) { +continue; +} for (int cmd = 0; cmd < 256; cmd++) { if (cxl_dstate->cxl_cmd_set[set][cmd].handler) { struct cxl_cmd *c = _dstate->cxl_cmd_set[set][cmd]; diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 329e8b5915b3..e6e6e125990c 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -1276,9 +1276,11 @@ static void ct3d_reset(DeviceState *dev) CXLType3Dev *ct3d = CXL_TYPE3(dev); uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers; uint32_t *write_msk = ct3d->cxl_cstate.crb.cache_mem_regs_write_mask; +bool is_dcd; cxl_component_register_init_common(reg_state, write_msk, CXL2_TYPE3_DEVICE); -cxl_device_register_init_common(>cxl_dstate); +is_dcd = (ct3d->dc.num_regions != 0); +cxl_device_register_init_common(>cxl_dstate, is_dcd); } static Property ct3_props[] = { diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 1ccddcca7d0d..4621bba4f533 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -233,7 +233,7 @@ typedef struct cxl_device_state { void cxl_device_register_block_init(Object *obj, CXLDeviceState *dev); /* Set up default values for the register block */ -void cxl_device_register_init_common(CXLDeviceState *dev); +void cxl_device_register_init_common(CXLDeviceState *dev, bool is_dcd); void cxl_device_register_init_swcci(CXLDeviceState *dev); /* @@ -280,7
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
The 06/08/2023 08:43, Ira Weiny wrote: > Shesha Bhushan Sreenivasamurthy wrote: > > Hi Fan, > >I am implementing DCD FMAPI commands and planning to start pushing > > changes to the below branch. That requires the contributions you have made. > > Can your changes be pushed to the below branch ? > > > > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25 > > This is the branch I'm trying to use as well. > > Thanks, > Ira Hi Ira & Shesha, FYI. I reabased my patch series on top of the above branch and created a new branch here: https://github.com/moking/qemu-dcd-preview-latest/tree/dcd-preview It passes the same tests as shown here: https://lore.kernel.org/linux-cxl/6481f70fca5c2_c82be29440@iweiny-mobl.notmuch/T/#m76f6e85ce3d7292b1982960eb22086ee03922166 -- Fan Ni
Re: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
Here is the repo where I am committing my changes https://gitlab.com/sheshas/qemu-fmapi/-/commits/cxl-2023-05-25 From: Shesha Bhushan Sreenivasamurthy Sent: Thursday, June 8, 2023 8:20 AM To: Jonathan Cameron Cc: Fan Ni ; qemu-devel@nongnu.org ; linux-...@vger.kernel.org ; gregory.pr...@memverge.com ; hch...@avery-design.com.tw ; cbr...@avery-design.com ; dan.j.willi...@intel.com ; Adam Manzanares ; d...@stgolabs.net ; nmtadam.sams...@gmail.com ; ni...@outlook.com ; Ira Weiny Subject: Re: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu From: Jonathan Cameron Sent: Thursday, June 8, 2023 2:43 AM To: Shesha Bhushan Sreenivasamurthy Cc: Fan Ni ; qemu-devel@nongnu.org ; linux-...@vger.kernel.org ; gregory.pr...@memverge.com ; hch...@avery-design.com.tw ; cbr...@avery-design.com ; dan.j.willi...@intel.com ; Adam Manzanares ; d...@stgolabs.net ; nmtadam.sams...@gmail.com ; ni...@outlook.com ; Ira Weiny Subject: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu External Email -- On Wed, 7 Jun 2023 18:52:14 + Shesha Bhushan Sreenivasamurthy wrote: > From: Fan Ni > Sent: Wednesday, June 7, 2023 11:31 AM > To: Shesha Bhushan Sreenivasamurthy > Cc: Jonathan Cameron ; qemu-devel@nongnu.org > ; linux-...@vger.kernel.org > ; gregory.pr...@memverge.com > ; hch...@avery-design.com.tw > ; cbr...@avery-design.com > ; dan.j.willi...@intel.com > ; Adam Manzanares ; > d...@stgolabs.net ; nmtadam.sams...@gmail.com > ; ni...@outlook.com ; Ira Weiny > > Subject: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > External Email > > -- > On Wed, Jun 07, 2023 at 06:13:01PM +, Shesha Bhushan Sreenivasamurthy > wrote: > > Hi Fan, > > I am implementing DCD FMAPI commands and planning to start pushing > >changes to the below branch. That requires the contributions you have made. > >Can your changes be pushed to the below branch ? > > > > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25__;!!EwVzqGoTKBqv-0DWAJBm!Vt5uIqwW-L4c4gh02ulI4M762JNQ3_aE9k9lb6QlwE2xm6T23ic7ig7Y77i1VN7l_RX_ySIQhre_z7Q0JA$ > > > > Can you push changes to the branch directly? I think it is Jonathan's private > branch. However, I can fork the branch and rebase my patch series atop and > share with you the new repo if that helps you move forward your > work. > Let me know your thought. > > ss - I saw commits from others, so assumed you can. Since it is Jonathan's > private repo, I will step back and let him answer. I tend to apply stuff in manually rather than given more people commit access to that particular gitlab tree. Easiest option is to fork on gitlab and share the path of your own fork. I normally queue reasonably mature stuff up on my tree, but that's about managing the series sent with intent of being applied upstream + providing a fairly stable test branch. It's not intended as a general place for stuff in development (though I might sneak out an extra branch myself from time to time if I want to talk about it :) Jonathan ss - Thanks ! I have forked Jonathan's repo on Gitlab and will work of cxl-2023-05-25 branch. I will apply Fan's patch manually and continue .. > > Fan > > > > > > > From: Fan Ni > > Sent: Monday, June 5, 2023 10:51 AM > > To: Ira Weiny > > Cc: qemu-devel@nongnu.org ; > > jonathan.came...@huawei.com ; > > linux-...@vger.kernel.org ; > > gregory.pr...@memverge.com ; > > hch...@avery-design.com.tw ; > > cbr...@avery-design.com ; dan.j.willi...@intel.com > > ; Adam Manzanares ; > > d...@stgolabs.net ; nmtadam.sams...@gmail.com > > ; ni...@outlook.com > > Subject: Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > > > On Mon, Jun 05, 2023 at 10:35:48AM -0700, Ira Weiny wrote: > > > Fan Ni wrote: > > > > Since the early draft of DCD support in kernel is out > > > > (https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21LzAGNOk$ > > > > ), > > > > this patch series provide dcd emulation in qemu so people who are > > > > interested > > > > can have an early try. It is noted that the patch series may need to be > > > > updated > > > > accordingly if the kernel side implementation changes. > > > > > > Fan, > > > > > > Do you have a git tree we can pull this from whic
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
Shesha Bhushan Sreenivasamurthy wrote: > Hi Fan, >I am implementing DCD FMAPI commands and planning to start pushing changes > to the below branch. That requires the contributions you have made. Can your > changes be pushed to the below branch ? > > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25 This is the branch I'm trying to use as well. Thanks, Ira
Re: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
From: Jonathan Cameron Sent: Thursday, June 8, 2023 2:43 AM To: Shesha Bhushan Sreenivasamurthy Cc: Fan Ni ; qemu-devel@nongnu.org ; linux-...@vger.kernel.org ; gregory.pr...@memverge.com ; hch...@avery-design.com.tw ; cbr...@avery-design.com ; dan.j.willi...@intel.com ; Adam Manzanares ; d...@stgolabs.net ; nmtadam.sams...@gmail.com ; ni...@outlook.com ; Ira Weiny Subject: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu External Email -- On Wed, 7 Jun 2023 18:52:14 + Shesha Bhushan Sreenivasamurthy wrote: > From: Fan Ni > Sent: Wednesday, June 7, 2023 11:31 AM > To: Shesha Bhushan Sreenivasamurthy > Cc: Jonathan Cameron ; qemu-devel@nongnu.org > ; linux-...@vger.kernel.org > ; gregory.pr...@memverge.com > ; hch...@avery-design.com.tw > ; cbr...@avery-design.com > ; dan.j.willi...@intel.com > ; Adam Manzanares ; > d...@stgolabs.net ; nmtadam.sams...@gmail.com > ; ni...@outlook.com ; Ira Weiny > > Subject: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > External Email > > -- > On Wed, Jun 07, 2023 at 06:13:01PM +, Shesha Bhushan Sreenivasamurthy > wrote: > > Hi Fan, > > I am implementing DCD FMAPI commands and planning to start pushing > >changes to the below branch. That requires the contributions you have made. > >Can your changes be pushed to the below branch ? > > > > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25__;!!EwVzqGoTKBqv-0DWAJBm!Vt5uIqwW-L4c4gh02ulI4M762JNQ3_aE9k9lb6QlwE2xm6T23ic7ig7Y77i1VN7l_RX_ySIQhre_z7Q0JA$ > > > > Can you push changes to the branch directly? I think it is Jonathan's private > branch. However, I can fork the branch and rebase my patch series atop and > share with you the new repo if that helps you move forward your > work. > Let me know your thought. > > ss - I saw commits from others, so assumed you can. Since it is Jonathan's > private repo, I will step back and let him answer. I tend to apply stuff in manually rather than given more people commit access to that particular gitlab tree. Easiest option is to fork on gitlab and share the path of your own fork. I normally queue reasonably mature stuff up on my tree, but that's about managing the series sent with intent of being applied upstream + providing a fairly stable test branch. It's not intended as a general place for stuff in development (though I might sneak out an extra branch myself from time to time if I want to talk about it :) Jonathan ss - Thanks ! I have forked Jonathan's repo on Gitlab and will work of cxl-2023-05-25 branch. I will apply Fan's patch manually and continue .. > > Fan > > > > > > > From: Fan Ni > > Sent: Monday, June 5, 2023 10:51 AM > > To: Ira Weiny > > Cc: qemu-devel@nongnu.org ; > > jonathan.came...@huawei.com ; > > linux-...@vger.kernel.org ; > > gregory.pr...@memverge.com ; > > hch...@avery-design.com.tw ; > > cbr...@avery-design.com ; dan.j.willi...@intel.com > > ; Adam Manzanares ; > > d...@stgolabs.net ; nmtadam.sams...@gmail.com > > ; ni...@outlook.com > > Subject: Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > > > On Mon, Jun 05, 2023 at 10:35:48AM -0700, Ira Weiny wrote: > > > Fan Ni wrote: > > > > Since the early draft of DCD support in kernel is out > > > > (https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21LzAGNOk$ > > > > ), > > > > this patch series provide dcd emulation in qemu so people who are > > > > interested > > > > can have an early try. It is noted that the patch series may need to be > > > > updated > > > > accordingly if the kernel side implementation changes. > > > > > > Fan, > > > > > > Do you have a git tree we can pull this from which is updated to a more > > > recent CXL branch from Jonathan? > > > > > > Thanks, > > > Ira > > > > Hi Ira, > > > > I have a git tree of the patch series based on Jonathan's branch > > cxl-2023-02-28: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_moking_qemu-2Ddev_tree_dcd-2Drfe=DwIFAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=w6dicn5kXEG4Imk6TpICIjdA6KJ-xt84dtHui-Y0fv5H13bijtzEvjxECKE5MHYf=3yeO9RN5FY3gPfO2y19X057YeqRTTQTQNfNA-Gfir_Q= > > . &g
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Wed, 7 Jun 2023 18:52:14 + Shesha Bhushan Sreenivasamurthy wrote: > From: Fan Ni > Sent: Wednesday, June 7, 2023 11:31 AM > To: Shesha Bhushan Sreenivasamurthy > Cc: Jonathan Cameron ; qemu-devel@nongnu.org > ; linux-...@vger.kernel.org > ; gregory.pr...@memverge.com > ; hch...@avery-design.com.tw > ; cbr...@avery-design.com > ; dan.j.willi...@intel.com > ; Adam Manzanares ; > d...@stgolabs.net ; nmtadam.sams...@gmail.com > ; ni...@outlook.com ; Ira Weiny > > Subject: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > External Email > > -- > On Wed, Jun 07, 2023 at 06:13:01PM +, Shesha Bhushan Sreenivasamurthy > wrote: > > Hi Fan, > > I am implementing DCD FMAPI commands and planning to start pushing > >changes to the below branch. That requires the contributions you have made. > >Can your changes be pushed to the below branch ? > > > > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25__;!!EwVzqGoTKBqv-0DWAJBm!Vt5uIqwW-L4c4gh02ulI4M762JNQ3_aE9k9lb6QlwE2xm6T23ic7ig7Y77i1VN7l_RX_ySIQhre_z7Q0JA$ > > > > Can you push changes to the branch directly? I think it is Jonathan's private > branch. However, I can fork the branch and rebase my patch series atop and > share with you the new repo if that helps you move forward your > work. > Let me know your thought. > > ss - I saw commits from others, so assumed you can. Since it is Jonathan's > private repo, I will step back and let him answer. I tend to apply stuff in manually rather than given more people commit access to that particular gitlab tree. Easiest option is to fork on gitlab and share the path of your own fork. I normally queue reasonably mature stuff up on my tree, but that's about managing the series sent with intent of being applied upstream + providing a fairly stable test branch. It's not intended as a general place for stuff in development (though I might sneak out an extra branch myself from time to time if I want to talk about it :) Jonathan > > Fan > > > > > > > From: Fan Ni > > Sent: Monday, June 5, 2023 10:51 AM > > To: Ira Weiny > > Cc: qemu-devel@nongnu.org ; > > jonathan.came...@huawei.com ; > > linux-...@vger.kernel.org ; > > gregory.pr...@memverge.com ; > > hch...@avery-design.com.tw ; > > cbr...@avery-design.com ; dan.j.willi...@intel.com > > ; Adam Manzanares ; > > d...@stgolabs.net ; nmtadam.sams...@gmail.com > > ; ni...@outlook.com > > Subject: Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > > > On Mon, Jun 05, 2023 at 10:35:48AM -0700, Ira Weiny wrote: > > > Fan Ni wrote: > > > > Since the early draft of DCD support in kernel is out > > > > (https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21LzAGNOk$ > > > > ), > > > > this patch series provide dcd emulation in qemu so people who are > > > > interested > > > > can have an early try. It is noted that the patch series may need to be > > > > updated > > > > accordingly if the kernel side implementation changes. > > > > > > Fan, > > > > > > Do you have a git tree we can pull this from which is updated to a more > > > recent CXL branch from Jonathan? > > > > > > Thanks, > > > Ira > > > > Hi Ira, > > > > I have a git tree of the patch series based on Jonathan's branch > > cxl-2023-02-28: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_moking_qemu-2Ddev_tree_dcd-2Drfe=DwIFAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=w6dicn5kXEG4Imk6TpICIjdA6KJ-xt84dtHui-Y0fv5H13bijtzEvjxECKE5MHYf=3yeO9RN5FY3gPfO2y19X057YeqRTTQTQNfNA-Gfir_Q= > > . > > > > That may be not new enough to include some of the recent patches, but I can > > rebase it to a newer branch if you can tell me which branch you want to use. > > > > Thanks, > > Fan > > > > > > > > > > > > > To support DCD emulation, the patch series add DCD related mailbox > > > > command > > > > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory > > > > device > > > > with dynamic capacity extent and region representative. > > > > To support read/write to the dynamic capacity of the device, a host > > > > backend &
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
Hi Fan, I am implementing DCD FMAPI commands and planning to start pushing changes to the below branch. That requires the contributions you have made. Can your changes be pushed to the below branch ? https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25 From: Fan Ni Sent: Monday, June 5, 2023 10:51 AM To: Ira Weiny Cc: qemu-devel@nongnu.org ; jonathan.came...@huawei.com ; linux-...@vger.kernel.org ; gregory.pr...@memverge.com ; hch...@avery-design.com.tw ; cbr...@avery-design.com ; dan.j.willi...@intel.com ; Adam Manzanares ; d...@stgolabs.net ; nmtadam.sams...@gmail.com ; ni...@outlook.com Subject: Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu On Mon, Jun 05, 2023 at 10:35:48AM -0700, Ira Weiny wrote: > Fan Ni wrote: > > Since the early draft of DCD support in kernel is out > > (https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21LzAGNOk$ > > ), > > this patch series provide dcd emulation in qemu so people who are interested > > can have an early try. It is noted that the patch series may need to be > > updated > > accordingly if the kernel side implementation changes. > > Fan, > > Do you have a git tree we can pull this from which is updated to a more > recent CXL branch from Jonathan? > > Thanks, > Ira Hi Ira, I have a git tree of the patch series based on Jonathan's branch cxl-2023-02-28: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_moking_qemu-2Ddev_tree_dcd-2Drfe=DwIFAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=w6dicn5kXEG4Imk6TpICIjdA6KJ-xt84dtHui-Y0fv5H13bijtzEvjxECKE5MHYf=3yeO9RN5FY3gPfO2y19X057YeqRTTQTQNfNA-Gfir_Q= . That may be not new enough to include some of the recent patches, but I can rebase it to a newer branch if you can tell me which branch you want to use. Thanks, Fan > > > > > To support DCD emulation, the patch series add DCD related mailbox command > > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device > > with dynamic capacity extent and region representative. > > To support read/write to the dynamic capacity of the device, a host backend > > is provided and necessary check mechnism is added to ensure the dynamic > > capacity accessed is backed with active dc extents. > > Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not > > supported > > , but we add two qmp interfaces for adding/releasing dynamic capacity > > extents. > > Also, the support for multiple hosts sharing the same DCD case is missing. > > > > Things we can try with the patch series together with kernel dcd code: > > 1. Create DC regions to cover the address range of the dynamic capacity > > regions. > > 2. Add/release dynamic capacity extents to the device and notify the > > kernel. > > 3. Test kernel side code to accept added dc extents and create dax devices, > > and release dc extents and notify the device > > 4. Online the memory range backed with dc extents and let application use > > them. > > > > The patch series is based on Jonathan's local qemu branch: > > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28__;!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21OO3UHEM$ > > > > > > Simple tests peformed with the patch series: > > 1 Install cxl modules: > > > > modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem > > > > 2 Create dc regions: > > > > region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) > > echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region > > echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity > > echo 1 > /sys/bus/cxl/devices/$region/interleave_ways > > echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode > > echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size > > echo 0x1000 > /sys/bus/cxl/devices/$region/size > > echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 > > echo 1 > /sys/bus/cxl/devices/$region/commit > > echo $region > /sys/bus/cxl/drivers/cxl_region/bind > > > > /home/fan/cxl/tools-and-scripts# cxl list > > [ > > { > > "memdevs":[ > > { > > "memdev":"mem0", > > "pmem_size":536870912, > > "ram_size":0, > > "serial":0, > > "host":":0d:00.0" > > } > >
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Wed, Jun 07, 2023 at 06:13:01PM +, Shesha Bhushan Sreenivasamurthy wrote: > Hi Fan, >I am implementing DCD FMAPI commands and planning to start pushing changes > to the below branch. That requires the contributions you have made. Can your > changes be pushed to the below branch ? > > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25__;!!EwVzqGoTKBqv-0DWAJBm!Vt5uIqwW-L4c4gh02ulI4M762JNQ3_aE9k9lb6QlwE2xm6T23ic7ig7Y77i1VN7l_RX_ySIQhre_z7Q0JA$ > Can you push changes to the branch directly? I think it is Jonathan's private branch. However, I can fork the branch and rebase my patch series atop and share with you the new repo if that helps you move forward your work. Let me know your thought. Fan > > > From: Fan Ni > Sent: Monday, June 5, 2023 10:51 AM > To: Ira Weiny > Cc: qemu-devel@nongnu.org ; > jonathan.came...@huawei.com ; > linux-...@vger.kernel.org ; > gregory.pr...@memverge.com ; > hch...@avery-design.com.tw ; > cbr...@avery-design.com ; dan.j.willi...@intel.com > ; Adam Manzanares ; > d...@stgolabs.net ; nmtadam.sams...@gmail.com > ; ni...@outlook.com > Subject: Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > On Mon, Jun 05, 2023 at 10:35:48AM -0700, Ira Weiny wrote: > > Fan Ni wrote: > > > Since the early draft of DCD support in kernel is out > > > (https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21LzAGNOk$ > > > ), > > > this patch series provide dcd emulation in qemu so people who are > > > interested > > > can have an early try. It is noted that the patch series may need to be > > > updated > > > accordingly if the kernel side implementation changes. > > > > Fan, > > > > Do you have a git tree we can pull this from which is updated to a more > > recent CXL branch from Jonathan? > > > > Thanks, > > Ira > > Hi Ira, > > I have a git tree of the patch series based on Jonathan's branch > cxl-2023-02-28: > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_moking_qemu-2Ddev_tree_dcd-2Drfe=DwIFAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=w6dicn5kXEG4Imk6TpICIjdA6KJ-xt84dtHui-Y0fv5H13bijtzEvjxECKE5MHYf=3yeO9RN5FY3gPfO2y19X057YeqRTTQTQNfNA-Gfir_Q= > . > > That may be not new enough to include some of the recent patches, but I can > rebase it to a newer branch if you can tell me which branch you want to use. > > Thanks, > Fan > > > > > > > > > To support DCD emulation, the patch series add DCD related mailbox command > > > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device > > > with dynamic capacity extent and region representative. > > > To support read/write to the dynamic capacity of the device, a host > > > backend > > > is provided and necessary check mechnism is added to ensure the dynamic > > > capacity accessed is backed with active dc extents. > > > Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not > > > supported > > > , but we add two qmp interfaces for adding/releasing dynamic capacity > > > extents. > > > Also, the support for multiple hosts sharing the same DCD case is missing. > > > > > > Things we can try with the patch series together with kernel dcd code: > > > 1. Create DC regions to cover the address range of the dynamic capacity > > > regions. > > > 2. Add/release dynamic capacity extents to the device and notify the > > > kernel. > > > 3. Test kernel side code to accept added dc extents and create dax > > > devices, > > > and release dc extents and notify the device > > > 4. Online the memory range backed with dc extents and let application use > > > them. > > > > > > The patch series is based on Jonathan's local qemu branch: > > > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28__;!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21OO3UHEM$ > > > > > > > > > Simple tests peformed with the patch series: > > > 1 Install cxl modules: > > > > > > modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem > > > > > > 2 Create dc regions: > > > > > > region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) > > > echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region > > > ec
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Mon, Jun 05, 2023 at 10:35:48AM -0700, Ira Weiny wrote: > Fan Ni wrote: > > Since the early draft of DCD support in kernel is out > > (https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21LzAGNOk$ > > ), > > this patch series provide dcd emulation in qemu so people who are interested > > can have an early try. It is noted that the patch series may need to be > > updated > > accordingly if the kernel side implementation changes. > > Fan, > > Do you have a git tree we can pull this from which is updated to a more > recent CXL branch from Jonathan? > > Thanks, > Ira Hi Ira, I have a git tree of the patch series based on Jonathan's branch cxl-2023-02-28: https://github.com/moking/qemu-dev/tree/dcd-rfe. That may be not new enough to include some of the recent patches, but I can rebase it to a newer branch if you can tell me which branch you want to use. Thanks, Fan > > > > > To support DCD emulation, the patch series add DCD related mailbox command > > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device > > with dynamic capacity extent and region representative. > > To support read/write to the dynamic capacity of the device, a host backend > > is provided and necessary check mechnism is added to ensure the dynamic > > capacity accessed is backed with active dc extents. > > Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not > > supported > > , but we add two qmp interfaces for adding/releasing dynamic capacity > > extents. > > Also, the support for multiple hosts sharing the same DCD case is missing. > > > > Things we can try with the patch series together with kernel dcd code: > > 1. Create DC regions to cover the address range of the dynamic capacity > > regions. > > 2. Add/release dynamic capacity extents to the device and notify the > > kernel. > > 3. Test kernel side code to accept added dc extents and create dax devices, > > and release dc extents and notify the device > > 4. Online the memory range backed with dc extents and let application use > > them. > > > > The patch series is based on Jonathan's local qemu branch: > > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28__;!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21OO3UHEM$ > > > > > > Simple tests peformed with the patch series: > > 1 Install cxl modules: > > > > modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem > > > > 2 Create dc regions: > > > > region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) > > echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region > > echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity > > echo 1 > /sys/bus/cxl/devices/$region/interleave_ways > > echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode > > echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size > > echo 0x1000 > /sys/bus/cxl/devices/$region/size > > echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 > > echo 1 > /sys/bus/cxl/devices/$region/commit > > echo $region > /sys/bus/cxl/drivers/cxl_region/bind > > > > /home/fan/cxl/tools-and-scripts# cxl list > > [ > > { > > "memdevs":[ > > { > > "memdev":"mem0", > > "pmem_size":536870912, > > "ram_size":0, > > "serial":0, > > "host":":0d:00.0" > > } > > ] > > }, > > { > > "regions":[ > > { > > "region":"region0", > > "resource":45365592064, > > "size":268435456, > > "interleave_ways":1, > > "interleave_granularity":256, > > "decode_state":"commit" > > } > > ] > > } > > ] > > > > 3 Add two dc extents (128MB each) through qmp interface > > > > { "execute": "qmp_capabilities" } > > > > { "execute": "cxl-add-dynamic-capacity-event", > > "arguments": { > > "path": "/machine/peripheral/cxl-pmem0", > > "region-id" : 0, > > "num-extent": 2, > > "dpa":0, > > "extent-len": 128 > > } > > } > > > > /home/fan/cxl/tools-and-scripts# lsmem > > RANGE SIZE STATE REMOVABLE BLOCK > > 0x-0x7fff2G online yes0-15 > > 0x0001-0x00027fff6G online yes 32-79 > > 0x000a9000-0x000a9fff 256M offline 338-339 > > > > Memory block size: 128M > > Total online memory: 8G > > Total offline memory:256M > > > > > > 4.Online the momory with 'daxctl online-memory dax0.0' to online the memory > > > > /home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0 > > [ 230.730553] Fallback order for Node 0: 0 1 > > [ 230.730825] Fallback order for Node 1: 1 0 > > [ 230.730953] Built 2 zonelists, mobility grouping on. Total pages: > >
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
Fan Ni wrote: > Since the early draft of DCD support in kernel is out > (https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/#t), > this patch series provide dcd emulation in qemu so people who are interested > can have an early try. It is noted that the patch series may need to be > updated > accordingly if the kernel side implementation changes. Fan, Do you have a git tree we can pull this from which is updated to a more recent CXL branch from Jonathan? Thanks, Ira > > To support DCD emulation, the patch series add DCD related mailbox command > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device > with dynamic capacity extent and region representative. > To support read/write to the dynamic capacity of the device, a host backend > is provided and necessary check mechnism is added to ensure the dynamic > capacity accessed is backed with active dc extents. > Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not supported > , but we add two qmp interfaces for adding/releasing dynamic capacity extents. > Also, the support for multiple hosts sharing the same DCD case is missing. > > Things we can try with the patch series together with kernel dcd code: > 1. Create DC regions to cover the address range of the dynamic capacity > regions. > 2. Add/release dynamic capacity extents to the device and notify the > kernel. > 3. Test kernel side code to accept added dc extents and create dax devices, > and release dc extents and notify the device > 4. Online the memory range backed with dc extents and let application use > them. > > The patch series is based on Jonathan's local qemu branch: > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28 > > Simple tests peformed with the patch series: > 1 Install cxl modules: > > modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem > > 2 Create dc regions: > > region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) > echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region > echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity > echo 1 > /sys/bus/cxl/devices/$region/interleave_ways > echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode > echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size > echo 0x1000 > /sys/bus/cxl/devices/$region/size > echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 > echo 1 > /sys/bus/cxl/devices/$region/commit > echo $region > /sys/bus/cxl/drivers/cxl_region/bind > > /home/fan/cxl/tools-and-scripts# cxl list > [ > { > "memdevs":[ > { > "memdev":"mem0", > "pmem_size":536870912, > "ram_size":0, > "serial":0, > "host":":0d:00.0" > } > ] > }, > { > "regions":[ > { > "region":"region0", > "resource":45365592064, > "size":268435456, > "interleave_ways":1, > "interleave_granularity":256, > "decode_state":"commit" > } > ] > } > ] > > 3 Add two dc extents (128MB each) through qmp interface > > { "execute": "qmp_capabilities" } > > { "execute": "cxl-add-dynamic-capacity-event", > "arguments": { >"path": "/machine/peripheral/cxl-pmem0", > "region-id" : 0, >"num-extent": 2, > "dpa":0, > "extent-len": 128 > } > } > > /home/fan/cxl/tools-and-scripts# lsmem > RANGE SIZE STATE REMOVABLE BLOCK > 0x-0x7fff2G online yes0-15 > 0x0001-0x00027fff6G online yes 32-79 > 0x000a9000-0x000a9fff 256M offline 338-339 > > Memory block size: 128M > Total online memory: 8G > Total offline memory:256M > > > 4.Online the momory with 'daxctl online-memory dax0.0' to online the memory > > /home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0 > [ 230.730553] Fallback order for Node 0: 0 1 > [ 230.730825] Fallback order for Node 1: 1 0 > [ 230.730953] Built 2 zonelists, mobility grouping on. Total pages: 2042541 > [ 230.731110] Policy zone: Normal > onlined memory for 1 device > > root@bgt-140510-bm03:/home/fan/cxl/ndctl# lsmem > RANGE SIZE STATE REMOVABLE BLOCK > 0x-0x7fff2G online yes 0-15 > 0x0001-0x00027fff6G online yes 32-79 > 0x000a9000-0x000a97ff 128M online yes 338 > 0x000a9800-0x000a9fff 128M offline 339 > > Memory block size: 128M > Total online memory: 8.1G > Total offline memory:128M > > 5 using dc extents as regular memory > > /home/fan/cxl/ndctl# numactl --membind=1 ls > CONTRIBUTING.md README.md clean_config.sh cscope.out git-version-gen > ndctlscripts test.h version.h.in COPYING acpi.h > config.h.meson cxlmake-git-snapshot.sh ndctl.spec.in sles
RE: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
-Original Message- From: Jonathan Cameron Sent: Monday, May 15, 2023 6:31 PM To: Fan Ni Cc: qemu-devel@nongnu.org; linux-...@vger.kernel.org; gregory.pr...@memverge.com; hch...@avery-design.com.tw; Browy, Christopher ; Weiny, Ira ; Williams, Dan J ; Adam Manzanares ; d...@stgolabs.net; nmtadam.sams...@gmail.com; ni...@outlook.com; Singh, Navneet Subject: Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu On Thu, 11 May 2023 17:56:40 + Fan Ni wrote: > Since the early draft of DCD support in kernel is out > (https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510 > -bm03/T/#t), this patch series provide dcd emulation in qemu so people > who are interested can have an early try. It is noted that the patch > series may need to be updated accordingly if the kernel side > implementation changes. > > To support DCD emulation, the patch series add DCD related mailbox > command support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 > memory device with dynamic capacity extent and region representative. > To support read/write to the dynamic capacity of the device, a host > backend is provided and necessary check mechnism is added to ensure > the dynamic capacity accessed is backed with active dc extents. > Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not > supported , but we add two qmp interfaces for adding/releasing dynamic > capacity extents. > Also, the support for multiple hosts sharing the same DCD case is missing. > > Things we can try with the patch series together with kernel dcd code: > 1. Create DC regions to cover the address range of the dynamic > capacity regions. > 2. Add/release dynamic capacity extents to the device and notify the > kernel. > 3. Test kernel side code to accept added dc extents and create dax > devices, and release dc extents and notify the device 4. Online the > memory range backed with dc extents and let application use them. > > The patch series is based on Jonathan's local qemu branch: > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28 > > Simple tests peformed with the patch series: > 1 Install cxl modules: > > modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem > > 2 Create dc regions: > > region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) > echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region > echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity > echo 1 > /sys/bus/cxl/devices/$region/interleave_ways > echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode > echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size > echo 0x1000 > /sys/bus/cxl/devices/$region/size echo "decoder2.0" > > /sys/bus/cxl/devices/$region/target0 > echo 1 > /sys/bus/cxl/devices/$region/commit > echo $region > /sys/bus/cxl/drivers/cxl_region/bind > > /home/fan/cxl/tools-and-scripts# cxl list [ > { > "memdevs":[ > { > "memdev":"mem0", > "pmem_size":536870912, > "ram_size":0, > "serial":0, > "host":":0d:00.0" > } > ] > }, > { > "regions":[ > { > "region":"region0", > "resource":45365592064, > "size":268435456, > "interleave_ways":1, > "interleave_granularity":256, > "decode_state":"commit" > } > ] > } > ] > > 3 Add two dc extents (128MB each) through qmp interface > > { "execute": "qmp_capabilities" } > > { "execute": "cxl-add-dynamic-capacity-event", > "arguments": { >"path": "/machine/peripheral/cxl-pmem0", > "region-id" : 0, >"num-extent": 2, > "dpa":0, > "extent-len": 128 > } > } > > /home/fan/cxl/tools-and-scripts# lsmem > RANGE SIZE STATE REMOVABLE BLOCK > 0x-0x7fff2G online yes0-15 > 0x0001-0x00027fff6G online yes 32-79 > 0x000a9000-0x000a9fff 256M offline 338-339 > > Memory block size: 128M > Total online memory: 8G > Total offline memory:256M > > > 4.Online the momory with 'daxctl online-memory dax0.0' to online the > memory > > /home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0 [ > 230.730553] Fallback order for Node 0: 0 1 [ 230.73
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Thu, 11 May 2023 17:56:40 + Fan Ni wrote: > Since the early draft of DCD support in kernel is out > (https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/#t), > this patch series provide dcd emulation in qemu so people who are interested > can have an early try. It is noted that the patch series may need to be > updated > accordingly if the kernel side implementation changes. > > To support DCD emulation, the patch series add DCD related mailbox command > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device > with dynamic capacity extent and region representative. > To support read/write to the dynamic capacity of the device, a host backend > is provided and necessary check mechnism is added to ensure the dynamic > capacity accessed is backed with active dc extents. > Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not supported > , but we add two qmp interfaces for adding/releasing dynamic capacity extents. > Also, the support for multiple hosts sharing the same DCD case is missing. > > Things we can try with the patch series together with kernel dcd code: > 1. Create DC regions to cover the address range of the dynamic capacity > regions. > 2. Add/release dynamic capacity extents to the device and notify the > kernel. > 3. Test kernel side code to accept added dc extents and create dax devices, > and release dc extents and notify the device > 4. Online the memory range backed with dc extents and let application use > them. > > The patch series is based on Jonathan's local qemu branch: > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28 > > Simple tests peformed with the patch series: > 1 Install cxl modules: > > modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem > > 2 Create dc regions: > > region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) > echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region > echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity > echo 1 > /sys/bus/cxl/devices/$region/interleave_ways > echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode > echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size > echo 0x1000 > /sys/bus/cxl/devices/$region/size > echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 > echo 1 > /sys/bus/cxl/devices/$region/commit > echo $region > /sys/bus/cxl/drivers/cxl_region/bind > > /home/fan/cxl/tools-and-scripts# cxl list > [ > { > "memdevs":[ > { > "memdev":"mem0", > "pmem_size":536870912, > "ram_size":0, > "serial":0, > "host":":0d:00.0" > } > ] > }, > { > "regions":[ > { > "region":"region0", > "resource":45365592064, > "size":268435456, > "interleave_ways":1, > "interleave_granularity":256, > "decode_state":"commit" > } > ] > } > ] > > 3 Add two dc extents (128MB each) through qmp interface > > { "execute": "qmp_capabilities" } > > { "execute": "cxl-add-dynamic-capacity-event", > "arguments": { >"path": "/machine/peripheral/cxl-pmem0", > "region-id" : 0, >"num-extent": 2, > "dpa":0, > "extent-len": 128 > } > } > > /home/fan/cxl/tools-and-scripts# lsmem > RANGE SIZE STATE REMOVABLE BLOCK > 0x-0x7fff2G online yes0-15 > 0x0001-0x00027fff6G online yes 32-79 > 0x000a9000-0x000a9fff 256M offline 338-339 > > Memory block size: 128M > Total online memory: 8G > Total offline memory:256M > > > 4.Online the momory with 'daxctl online-memory dax0.0' to online the memory > > /home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0 > [ 230.730553] Fallback order for Node 0: 0 1 > [ 230.730825] Fallback order for Node 1: 1 0 > [ 230.730953] Built 2 zonelists, mobility grouping on. Total pages: 2042541 > [ 230.731110] Policy zone: Normal > onlined memory for 1 device > > root@bgt-140510-bm03:/home/fan/cxl/ndctl# lsmem > RANGE SIZE STATE REMOVABLE BLOCK > 0x-0x7fff2G online yes 0-15 > 0x0001-0x00027fff6G online yes 32-79 > 0x000a9000-0x000a97ff 128M online yes 338 > 0x000a9800-0x000a9fff 128M offline 339 > > Memory block size: 128M > Total online memory: 8.1G > Total offline memory:128M > > 5 using dc extents as regular memory > > /home/fan/cxl/ndctl# numactl --membind=1 ls > CONTRIBUTING.md README.md clean_config.sh cscope.out git-version-gen > ndctlscripts test.h version.h.in COPYING acpi.h > config.h.meson cxlmake-git-snapshot.sh ndctl.spec.in sles tools > Documentation build contrib daxctl meson.build >