Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Tue, Jul 25, 2023 at 08:18:08AM -0700, Ira Weiny wrote:
> Fan Ni wrote:
> > On Thu, May 11, 2023 at 05:56:40PM +, Fan Ni wrote:
> >
> > FYI.
> >
> > I have updated the patch series and sent out again.
> >
> > I suggested anyone who are interested in DCD and using this patch series to
> > use the new series. Quite a few things has been fixed.
> >
> > https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230724162313.34196-1-fan...@samsung.com/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!V8kDTpT5yLUAyWm3sFm7XIgN0QdNUyQYZd9vYLHjUVkMkhDT14F8avgNBh23KPAtsS_dGm2LZuHJ102mgIg$
> >
> >
> > Also, if you want to use the code repo directly, you can try
> >
> > https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=0600fd6b-678be820-06017624-74fe485fb305-f529279062b02b73=1=5df65010-e62f-40a1-9a21-609eb1400921=https*3A*2F*2Fgithub.com*2Fmoking*2Fqemu-dcd-preview-latest*2Ftree*2Fdcd-dev__;JSUlJSUlJQ!!EwVzqGoTKBqv-0DWAJBm!V8kDTpT5yLUAyWm3sFm7XIgN0QdNUyQYZd9vYLHjUVkMkhDT14F8avgNBh23KPAtsS_dGm2LZuHJpB0SmNs$
> >
>
> Thanks for the branch!
>
> I took a quick look and I don't see a resolution to the problem I
> mentioned with non DCD devices being supported.[1]
>
> [1]
> https://urldefense.com/v3/__https://lore.kernel.org/all/6483946e8152f_f1132294a2@iweiny-mobl.notmuch/__;!!EwVzqGoTKBqv-0DWAJBm!V8kDTpT5yLUAyWm3sFm7XIgN0QdNUyQYZd9vYLHjUVkMkhDT14F8avgNBh23KPAtsS_dGm2LZuHJ4geSEAg$
>
>
> Did you fix this in a different way? If I don't add DC to my mem devices they
> don't get probed properly. I'm still looking into this with your new branch,
> but I don't think DC commands should be in the CEL if the device does not
> support it.
>
> Also I get a build warning on this branch I had to fix[3] as my build is
> treating warnings as errors.[2]
>
> I don't think this fix is technically necessary as 'list' should never be NULL
> that I can see. But might be nice to check or just use my fix.
>
> I'll try and get to a review once I get the DCD stuff out on the list again.
>
> Ira
>
Oh, I missed your previous comments, let me look into it and fix
accordingly and send out a new version.
Btw, when I did the DCD test with the last DCD kernel code, I found
some issue there.
When I add a DCD extent for the first time, it will be recognized as
system RAM automatically and show up with command lsmem.
However, when I release it and try to re-add the same extent again.
The adding seems normal and the device will show up under /dev/ as
dax0.X. But it will not show up with lsmem command and I have to use
daxctl reconfigure command to turn it to system ram and then it can
show up with lsmem command. I would expect the behavior for the
first add and second add be the same.
Fan.
>
> [2]
> ../hw/mem/cxl_type3.c: In function
> ‘qmp_cxl_process_dynamic_capacity_event.constprop’:
> ../hw/mem/cxl_type3.c:2063:28: error: ‘rid’ may be used uninitialized
> [-Werror=maybe-uninitialized]
> 2063 | dCap.updated_region_id = rid;
> | ~~~^
> ../hw/mem/cxl_type3.c:1987:13: note: ‘rid’ was declared here
> 1987 | uint8_t rid;
> | ^~~
> cc1: all warnings being treated as errors
>
> [3]
>
> diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
> index e67328780407..d25e6064f6c9 100644
> --- a/hw/mem/cxl_type3.c
> +++ b/hw/mem/cxl_type3.c
> @@ -1984,7 +1984,7 @@ static void
> qmp_cxl_process_dynamic_capacity_event(const char *path,
> CXLDCExtentRecordList *list = records;
> CXLDCExtent_raw *extents;
> uint64_t dpa, len;
> -uint8_t rid;
> +uint8_t rid = 0;
> int i;
>
> if (!obj) {
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
Fan Ni wrote:
> On Thu, May 11, 2023 at 05:56:40PM +, Fan Ni wrote:
>
> FYI.
>
> I have updated the patch series and sent out again.
>
> I suggested anyone who are interested in DCD and using this patch series to
> use the new series. Quite a few things has been fixed.
>
> https://lore.kernel.org/linux-cxl/20230724162313.34196-1-fan...@samsung.com/T/#t
>
> Also, if you want to use the code repo directly, you can try
>
> https://github.com/moking/qemu-dcd-preview-latest/tree/dcd-dev
Thanks for the branch!
I took a quick look and I don't see a resolution to the problem I
mentioned with non DCD devices being supported.[1]
[1] https://lore.kernel.org/all/6483946e8152f_f1132294a2@iweiny-mobl.notmuch/
Did you fix this in a different way? If I don't add DC to my mem devices they
don't get probed properly. I'm still looking into this with your new branch,
but I don't think DC commands should be in the CEL if the device does not
support it.
Also I get a build warning on this branch I had to fix[3] as my build is
treating warnings as errors.[2]
I don't think this fix is technically necessary as 'list' should never be NULL
that I can see. But might be nice to check or just use my fix.
I'll try and get to a review once I get the DCD stuff out on the list again.
Ira
[2]
../hw/mem/cxl_type3.c: In function
‘qmp_cxl_process_dynamic_capacity_event.constprop’:
../hw/mem/cxl_type3.c:2063:28: error: ‘rid’ may be used uninitialized
[-Werror=maybe-uninitialized]
2063 | dCap.updated_region_id = rid;
| ~~~^
../hw/mem/cxl_type3.c:1987:13: note: ‘rid’ was declared here
1987 | uint8_t rid;
| ^~~
cc1: all warnings being treated as errors
[3]
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index e67328780407..d25e6064f6c9 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -1984,7 +1984,7 @@ static void qmp_cxl_process_dynamic_capacity_event(const
char *path,
CXLDCExtentRecordList *list = records;
CXLDCExtent_raw *extents;
uint64_t dpa, len;
-uint8_t rid;
+uint8_t rid = 0;
int i;
if (!obj) {
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Thu, May 11, 2023 at 05:56:40PM +, Fan Ni wrote:
FYI.
I have updated the patch series and sent out again.
I suggested anyone who are interested in DCD and using this patch series to
use the new series. Quite a few things has been fixed.
https://lore.kernel.org/linux-cxl/20230724162313.34196-1-fan...@samsung.com/T/#t
Also, if you want to use the code repo directly, you can try
https://github.com/moking/qemu-dcd-preview-latest/tree/dcd-dev
Fan
> Since the early draft of DCD support in kernel is out
> (https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/#t),
> this patch series provide dcd emulation in qemu so people who are interested
> can have an early try. It is noted that the patch series may need to be
> updated
> accordingly if the kernel side implementation changes.
>
> To support DCD emulation, the patch series add DCD related mailbox command
> support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device
> with dynamic capacity extent and region representative.
> To support read/write to the dynamic capacity of the device, a host backend
> is provided and necessary check mechnism is added to ensure the dynamic
> capacity accessed is backed with active dc extents.
> Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not supported
> , but we add two qmp interfaces for adding/releasing dynamic capacity extents.
> Also, the support for multiple hosts sharing the same DCD case is missing.
>
> Things we can try with the patch series together with kernel dcd code:
> 1. Create DC regions to cover the address range of the dynamic capacity
> regions.
> 2. Add/release dynamic capacity extents to the device and notify the
> kernel.
> 3. Test kernel side code to accept added dc extents and create dax devices,
> and release dc extents and notify the device
> 4. Online the memory range backed with dc extents and let application use
> them.
>
> The patch series is based on Jonathan's local qemu branch:
> https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28
>
> Simple tests peformed with the patch series:
> 1 Install cxl modules:
>
> modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem
>
> 2 Create dc regions:
>
> region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region)
> echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region
> echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity
> echo 1 > /sys/bus/cxl/devices/$region/interleave_ways
> echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode
> echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size
> echo 0x1000 > /sys/bus/cxl/devices/$region/size
> echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0
> echo 1 > /sys/bus/cxl/devices/$region/commit
> echo $region > /sys/bus/cxl/drivers/cxl_region/bind
>
> /home/fan/cxl/tools-and-scripts# cxl list
> [
> {
> "memdevs":[
> {
> "memdev":"mem0",
> "pmem_size":536870912,
> "ram_size":0,
> "serial":0,
> "host":":0d:00.0"
> }
> ]
> },
> {
> "regions":[
> {
> "region":"region0",
> "resource":45365592064,
> "size":268435456,
> "interleave_ways":1,
> "interleave_granularity":256,
> "decode_state":"commit"
> }
> ]
> }
> ]
>
> 3 Add two dc extents (128MB each) through qmp interface
>
> { "execute": "qmp_capabilities" }
>
> { "execute": "cxl-add-dynamic-capacity-event",
> "arguments": {
>"path": "/machine/peripheral/cxl-pmem0",
> "region-id" : 0,
>"num-extent": 2,
> "dpa":0,
> "extent-len": 128
> }
> }
>
> /home/fan/cxl/tools-and-scripts# lsmem
> RANGE SIZE STATE REMOVABLE BLOCK
> 0x-0x7fff2G online yes0-15
> 0x0001-0x00027fff6G online yes 32-79
> 0x000a9000-0x000a9fff 256M offline 338-339
>
> Memory block size: 128M
> Total online memory: 8G
> Total offline memory:256M
>
>
> 4.Online the momory with 'daxctl online-memory dax0.0' to online the memory
>
> /home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0
> [ 230.730553] Fallback order for Node 0: 0 1
> [ 230.730825] Fallback order for Node 1: 1 0
> [ 230.730953] Built 2 zonelists, mobility grouping on. Total pages: 2042541
> [ 230.731110] Policy zone: Normal
> onlined memory for 1 device
>
> root@bgt-140510-bm03:/home/fan/cxl/ndctl# lsmem
> RANGE SIZE STATE REMOVABLE BLOCK
> 0x-0x7fff2G online yes 0-15
> 0x0001-0x00027fff6G online yes 32-79
> 0x000a9000-0x000a97ff 128M online yes 338
> 0x000a9800-0x000a9fff 128M offline 339
>
> Memory block size: 128M
> Total online memory: 8.1G
> Total offline memory:
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
The 05/15/2023 14:00, Jonathan Cameron wrote:
> On Thu, 11 May 2023 17:56:40 +
> Fan Ni wrote:
>
> > Since the early draft of DCD support in kernel is out
> > (https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/#t),
> > this patch series provide dcd emulation in qemu so people who are interested
> > can have an early try. It is noted that the patch series may need to be
> > updated
> > accordingly if the kernel side implementation changes.
> >
> > To support DCD emulation, the patch series add DCD related mailbox command
> > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device
> > with dynamic capacity extent and region representative.
> > To support read/write to the dynamic capacity of the device, a host backend
> > is provided and necessary check mechnism is added to ensure the dynamic
> > capacity accessed is backed with active dc extents.
> > Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not
> > supported
> > , but we add two qmp interfaces for adding/releasing dynamic capacity
> > extents.
> > Also, the support for multiple hosts sharing the same DCD case is missing.
> >
> > Things we can try with the patch series together with kernel dcd code:
> > 1. Create DC regions to cover the address range of the dynamic capacity
> > regions.
> > 2. Add/release dynamic capacity extents to the device and notify the
> > kernel.
> > 3. Test kernel side code to accept added dc extents and create dax devices,
> > and release dc extents and notify the device
> > 4. Online the memory range backed with dc extents and let application use
> > them.
> >
> > The patch series is based on Jonathan's local qemu branch:
> > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28
> >
> > Simple tests peformed with the patch series:
> > 1 Install cxl modules:
> >
> > modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem
> >
> > 2 Create dc regions:
> >
> > region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region)
> > echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region
> > echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity
> > echo 1 > /sys/bus/cxl/devices/$region/interleave_ways
> > echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode
> > echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size
> > echo 0x1000 > /sys/bus/cxl/devices/$region/size
> > echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0
> > echo 1 > /sys/bus/cxl/devices/$region/commit
> > echo $region > /sys/bus/cxl/drivers/cxl_region/bind
> >
> > /home/fan/cxl/tools-and-scripts# cxl list
> > [
> > {
> > "memdevs":[
> > {
> > "memdev":"mem0",
> > "pmem_size":536870912,
> > "ram_size":0,
> > "serial":0,
> > "host":":0d:00.0"
> > }
> > ]
> > },
> > {
> > "regions":[
> > {
> > "region":"region0",
> > "resource":45365592064,
> > "size":268435456,
> > "interleave_ways":1,
> > "interleave_granularity":256,
> > "decode_state":"commit"
> > }
> > ]
> > }
> > ]
> >
> > 3 Add two dc extents (128MB each) through qmp interface
> >
> > { "execute": "qmp_capabilities" }
> >
> > { "execute": "cxl-add-dynamic-capacity-event",
> > "arguments": {
> > "path": "/machine/peripheral/cxl-pmem0",
> > "region-id" : 0,
> > "num-extent": 2,
> > "dpa":0,
> > "extent-len": 128
> > }
> > }
> >
> > /home/fan/cxl/tools-and-scripts# lsmem
> > RANGE SIZE STATE REMOVABLE BLOCK
> > 0x-0x7fff2G online yes0-15
> > 0x0001-0x00027fff6G online yes 32-79
> > 0x000a9000-0x000a9fff 256M offline 338-339
> >
> > Memory block size: 128M
> > Total online memory: 8G
> > Total offline memory:256M
> >
> >
> > 4.Online the momory with 'daxctl online-memory dax0.0' to online the memory
> >
> > /home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0
> > [ 230.730553] Fallback order for Node 0: 0 1
> > [ 230.730825] Fallback order for Node 1: 1 0
> > [ 230.730953] Built 2 zonelists, mobility grouping on. Total pages:
> > 2042541
> > [ 230.731110] Policy zone: Normal
> > onlined memory for 1 device
> >
> > root@bgt-140510-bm03:/home/fan/cxl/ndctl# lsmem
> > RANGE SIZE STATE REMOVABLE BLOCK
> > 0x-0x7fff2G online yes 0-15
> > 0x0001-0x00027fff6G online yes 32-79
> > 0x000a9000-0x000a97ff 128M online yes 338
> > 0x000a9800-0x000a9fff 128M offline 339
> >
> > Memory block size: 128M
> > Total online memory: 8.1G
> > Total offline memory:128M
> >
> > 5 using dc extents as regular memory
> >
> > /home/fan/cxl/ndctl# numactl --membind=1 ls
> >
Re: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
From: Ira Weiny
Sent: Friday, June 9, 2023 2:06 PM
To: ni...@outlook.com ; Ira Weiny ;
Shesha Bhushan Sreenivasamurthy
Cc: Shesha Bhushan Sreenivasamurthy ; Fan Ni
; Jonathan Cameron ;
qemu-devel@nongnu.org ; linux-...@vger.kernel.org
; gregory.pr...@memverge.com
; hch...@avery-design.com.tw
; cbr...@avery-design.com
; dan.j.willi...@intel.com ;
Adam Manzanares ; d...@stgolabs.net
; nmtadam.sams...@gmail.com
Subject: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
External Email
--
ni...@outlook.com wrote:
> The 06/08/2023 08:43, Ira Weiny wrote:
> > Shesha Bhushan Sreenivasamurthy wrote:
[snip]
>
> Hi Ira & Shesha,
> FYI. I reabased my patch series on top of the above branch and created a new
> branch here:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_moking_qemu-2Ddcd-2Dpreview-2Dlatest_tree_dcd-2Dpreview=DwIBAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=SvyB_49EIFUT8-ZEVgIEYYjU6-zGTX4wb30kuNLUhkTSHYZK5-C0Gxr7uvefhtj4=MFD7qlSaTuy-w6aDmavIMbSP_aeaqZmSML7IVOX5jLs=
>
Thanks!
>
> It passes the same tests as shown here:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_linux-2Dcxl_6481f70fca5c2-5Fc82be29440-40iweiny-2Dmobl.notmuch_T_-23m76f6e85ce3d7292b1982960eb22086ee03922166=DwIBAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=SvyB_49EIFUT8-ZEVgIEYYjU6-zGTX4wb30kuNLUhkTSHYZK5-C0Gxr7uvefhtj4=e-fQOi0RzSZXxfSz37Bpz1sKtp7Yy0MWqonZnswK0RU=
>
I've not gotten very far with this testing. But I did find that regular
type 3 devices don't work with this change. I used the patch below to get
this working. Was there something I was missing to configure a non-DCD
device?
I don't particularly like adding another bool to this call stack. Seems
like this calls for a flags field but I want to move on to DCD work so I
hacked this in.
I am working on the DCD FM-API commands here -
https://gitlab.com/sheshas/qemu-fmapi/-/tree/cxl-2023-05-25
-Shesha
Ira
commit ed27935044dcbd2c6ba71f8411b218621f3f4167
Author: Ira Weiny
Date: Fri Jun 9 13:56:33 2023 -0700
hw/mem/cxl_type3: Exclude DCD from CEL when type3 is not DCD
Per CXL 3.0 9.13.3 Dynamic Capacity Device (DCD) when the type 3 memory
device does not have DCD support the CEL should not include DCD
configuration commands.
If the number of DC regions supported is 0 skip the DCD commands in the
CEL.
Applies on top of Fan Ni's work here:
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_moking_qemu-2Ddcd-2Dpreview-2Dlatest_tree_dcd-2Dpreview=DwIBAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=SvyB_49EIFUT8-ZEVgIEYYjU6-zGTX4wb30kuNLUhkTSHYZK5-C0Gxr7uvefhtj4=MFD7qlSaTuy-w6aDmavIMbSP_aeaqZmSML7IVOX5jLs=
Not-yet-Signed-off-by: Ira Weiny
diff --git a/hw/cxl/cxl-device-utils.c b/hw/cxl/cxl-device-utils.c
index a4a2c6a80004..262e35935563 100644
--- a/hw/cxl/cxl-device-utils.c
+++ b/hw/cxl/cxl-device-utils.c
@@ -288,7 +288,7 @@ static void mailbox_reg_init_common(CXLDeviceState
*cxl_dstate)
static void memdev_reg_init_common(CXLDeviceState *cxl_dstate) { }
-void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
+void cxl_device_register_init_common(CXLDeviceState *cxl_dstate, bool is_dcd)
{
uint64_t *cap_hdrs = cxl_dstate->caps_reg_state64;
const int cap_count = 3;
@@ -307,7 +307,7 @@ void cxl_device_register_init_common(CXLDeviceState
*cxl_dstate)
cxl_device_cap_init(cxl_dstate, MEMORY_DEVICE, 0x4000, 1);
memdev_reg_init_common(cxl_dstate);
- cxl_initialize_mailbox(cxl_dstate, false);
+ cxl_initialize_mailbox(cxl_dstate, false, is_dcd);
}
void cxl_device_register_init_swcci(CXLDeviceState *cxl_dstate)
@@ -329,7 +329,7 @@ void cxl_device_register_init_swcci(CXLDeviceState
*cxl_dstate)
cxl_device_cap_init(cxl_dstate, MEMORY_DEVICE, 0x4000, 1);
memdev_reg_init_common(cxl_dstate);
- cxl_initialize_mailbox(cxl_dstate, true);
+ cxl_initialize_mailbox(cxl_dstate, true, false);
}
uint64_t cxl_device_get_timestamp(CXLDeviceState *cxl_dstate)
diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
index 93b26e717c94..80e9cb9a8f04 100644
--- a/hw/cxl/cxl-mailbox-utils.c
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -1526,7 +1526,8 @@ static void bg_timercb(void *opaque)
}
}
-void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate, bool switch_cci)
+void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate, bool switch_cci,
+ bool is_dcd)
{
if (!switch_cci) {
cxl_dstate->cxl_cmd_set = cxl_cmd_set;
@@ -1534,6 +1535,9 @@ void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate,
bool switch_cci)
cxl_dstate->cxl_cmd_set = cxl_cmd_set_sw;
}
for (int set = 0; set < 256; set++) {
+
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
ni...@outlook.com wrote:
> The 06/08/2023 08:43, Ira Weiny wrote:
> > Shesha Bhushan Sreenivasamurthy wrote:
[snip]
>
> Hi Ira & Shesha,
> FYI. I reabased my patch series on top of the above branch and created a new
> branch here:
>
> https://github.com/moking/qemu-dcd-preview-latest/tree/dcd-preview
Thanks!
>
> It passes the same tests as shown here:
> https://lore.kernel.org/linux-cxl/6481f70fca5c2_c82be29440@iweiny-mobl.notmuch/T/#m76f6e85ce3d7292b1982960eb22086ee03922166
I've not gotten very far with this testing. But I did find that regular
type 3 devices don't work with this change. I used the patch below to get
this working. Was there something I was missing to configure a non-DCD
device?
I don't particularly like adding another bool to this call stack. Seems
like this calls for a flags field but I want to move on to DCD work so I
hacked this in.
Ira
commit ed27935044dcbd2c6ba71f8411b218621f3f4167
Author: Ira Weiny
Date: Fri Jun 9 13:56:33 2023 -0700
hw/mem/cxl_type3: Exclude DCD from CEL when type3 is not DCD
Per CXL 3.0 9.13.3 Dynamic Capacity Device (DCD) when the type 3 memory
device does not have DCD support the CEL should not include DCD
configuration commands.
If the number of DC regions supported is 0 skip the DCD commands in the
CEL.
Applies on top of Fan Ni's work here:
https://github.com/moking/qemu-dcd-preview-latest/tree/dcd-preview
Not-yet-Signed-off-by: Ira Weiny
diff --git a/hw/cxl/cxl-device-utils.c b/hw/cxl/cxl-device-utils.c
index a4a2c6a80004..262e35935563 100644
--- a/hw/cxl/cxl-device-utils.c
+++ b/hw/cxl/cxl-device-utils.c
@@ -288,7 +288,7 @@ static void mailbox_reg_init_common(CXLDeviceState
*cxl_dstate)
static void memdev_reg_init_common(CXLDeviceState *cxl_dstate) { }
-void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
+void cxl_device_register_init_common(CXLDeviceState *cxl_dstate, bool is_dcd)
{
uint64_t *cap_hdrs = cxl_dstate->caps_reg_state64;
const int cap_count = 3;
@@ -307,7 +307,7 @@ void cxl_device_register_init_common(CXLDeviceState
*cxl_dstate)
cxl_device_cap_init(cxl_dstate, MEMORY_DEVICE, 0x4000, 1);
memdev_reg_init_common(cxl_dstate);
-cxl_initialize_mailbox(cxl_dstate, false);
+cxl_initialize_mailbox(cxl_dstate, false, is_dcd);
}
void cxl_device_register_init_swcci(CXLDeviceState *cxl_dstate)
@@ -329,7 +329,7 @@ void cxl_device_register_init_swcci(CXLDeviceState
*cxl_dstate)
cxl_device_cap_init(cxl_dstate, MEMORY_DEVICE, 0x4000, 1);
memdev_reg_init_common(cxl_dstate);
-cxl_initialize_mailbox(cxl_dstate, true);
+cxl_initialize_mailbox(cxl_dstate, true, false);
}
uint64_t cxl_device_get_timestamp(CXLDeviceState *cxl_dstate)
diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
index 93b26e717c94..80e9cb9a8f04 100644
--- a/hw/cxl/cxl-mailbox-utils.c
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -1526,7 +1526,8 @@ static void bg_timercb(void *opaque)
}
}
-void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate, bool switch_cci)
+void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate, bool switch_cci,
+bool is_dcd)
{
if (!switch_cci) {
cxl_dstate->cxl_cmd_set = cxl_cmd_set;
@@ -1534,6 +1535,9 @@ void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate,
bool switch_cci)
cxl_dstate->cxl_cmd_set = cxl_cmd_set_sw;
}
for (int set = 0; set < 256; set++) {
+if (!is_dcd && set == DCD_CONFIG) {
+continue;
+}
for (int cmd = 0; cmd < 256; cmd++) {
if (cxl_dstate->cxl_cmd_set[set][cmd].handler) {
struct cxl_cmd *c = _dstate->cxl_cmd_set[set][cmd];
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index 329e8b5915b3..e6e6e125990c 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -1276,9 +1276,11 @@ static void ct3d_reset(DeviceState *dev)
CXLType3Dev *ct3d = CXL_TYPE3(dev);
uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
uint32_t *write_msk = ct3d->cxl_cstate.crb.cache_mem_regs_write_mask;
+bool is_dcd;
cxl_component_register_init_common(reg_state, write_msk,
CXL2_TYPE3_DEVICE);
-cxl_device_register_init_common(>cxl_dstate);
+is_dcd = (ct3d->dc.num_regions != 0);
+cxl_device_register_init_common(>cxl_dstate, is_dcd);
}
static Property ct3_props[] = {
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index 1ccddcca7d0d..4621bba4f533 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -233,7 +233,7 @@ typedef struct cxl_device_state {
void cxl_device_register_block_init(Object *obj, CXLDeviceState *dev);
/* Set up default values for the register block */
-void cxl_device_register_init_common(CXLDeviceState *dev);
+void cxl_device_register_init_common(CXLDeviceState *dev, bool is_dcd);
void cxl_device_register_init_swcci(CXLDeviceState *dev);
/*
@@ -280,7
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
The 06/08/2023 08:43, Ira Weiny wrote: > Shesha Bhushan Sreenivasamurthy wrote: > > Hi Fan, > >I am implementing DCD FMAPI commands and planning to start pushing > > changes to the below branch. That requires the contributions you have made. > > Can your changes be pushed to the below branch ? > > > > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25 > > This is the branch I'm trying to use as well. > > Thanks, > Ira Hi Ira & Shesha, FYI. I reabased my patch series on top of the above branch and created a new branch here: https://github.com/moking/qemu-dcd-preview-latest/tree/dcd-preview It passes the same tests as shown here: https://lore.kernel.org/linux-cxl/6481f70fca5c2_c82be29440@iweiny-mobl.notmuch/T/#m76f6e85ce3d7292b1982960eb22086ee03922166 -- Fan Ni
Re: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
Here is the repo where I am committing my changes https://gitlab.com/sheshas/qemu-fmapi/-/commits/cxl-2023-05-25 From: Shesha Bhushan Sreenivasamurthy Sent: Thursday, June 8, 2023 8:20 AM To: Jonathan Cameron Cc: Fan Ni ; qemu-devel@nongnu.org ; linux-...@vger.kernel.org ; gregory.pr...@memverge.com ; hch...@avery-design.com.tw ; cbr...@avery-design.com ; dan.j.willi...@intel.com ; Adam Manzanares ; d...@stgolabs.net ; nmtadam.sams...@gmail.com ; ni...@outlook.com ; Ira Weiny Subject: Re: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu From: Jonathan Cameron Sent: Thursday, June 8, 2023 2:43 AM To: Shesha Bhushan Sreenivasamurthy Cc: Fan Ni ; qemu-devel@nongnu.org ; linux-...@vger.kernel.org ; gregory.pr...@memverge.com ; hch...@avery-design.com.tw ; cbr...@avery-design.com ; dan.j.willi...@intel.com ; Adam Manzanares ; d...@stgolabs.net ; nmtadam.sams...@gmail.com ; ni...@outlook.com ; Ira Weiny Subject: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu External Email -- On Wed, 7 Jun 2023 18:52:14 + Shesha Bhushan Sreenivasamurthy wrote: > From: Fan Ni > Sent: Wednesday, June 7, 2023 11:31 AM > To: Shesha Bhushan Sreenivasamurthy > Cc: Jonathan Cameron ; qemu-devel@nongnu.org > ; linux-...@vger.kernel.org > ; gregory.pr...@memverge.com > ; hch...@avery-design.com.tw > ; cbr...@avery-design.com > ; dan.j.willi...@intel.com > ; Adam Manzanares ; > d...@stgolabs.net ; nmtadam.sams...@gmail.com > ; ni...@outlook.com ; Ira Weiny > > Subject: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > External Email > > -- > On Wed, Jun 07, 2023 at 06:13:01PM +, Shesha Bhushan Sreenivasamurthy > wrote: > > Hi Fan, > > I am implementing DCD FMAPI commands and planning to start pushing > >changes to the below branch. That requires the contributions you have made. > >Can your changes be pushed to the below branch ? > > > > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25__;!!EwVzqGoTKBqv-0DWAJBm!Vt5uIqwW-L4c4gh02ulI4M762JNQ3_aE9k9lb6QlwE2xm6T23ic7ig7Y77i1VN7l_RX_ySIQhre_z7Q0JA$ > > > > Can you push changes to the branch directly? I think it is Jonathan's private > branch. However, I can fork the branch and rebase my patch series atop and > share with you the new repo if that helps you move forward your > work. > Let me know your thought. > > ss - I saw commits from others, so assumed you can. Since it is Jonathan's > private repo, I will step back and let him answer. I tend to apply stuff in manually rather than given more people commit access to that particular gitlab tree. Easiest option is to fork on gitlab and share the path of your own fork. I normally queue reasonably mature stuff up on my tree, but that's about managing the series sent with intent of being applied upstream + providing a fairly stable test branch. It's not intended as a general place for stuff in development (though I might sneak out an extra branch myself from time to time if I want to talk about it :) Jonathan ss - Thanks ! I have forked Jonathan's repo on Gitlab and will work of cxl-2023-05-25 branch. I will apply Fan's patch manually and continue .. > > Fan > > > > > > > From: Fan Ni > > Sent: Monday, June 5, 2023 10:51 AM > > To: Ira Weiny > > Cc: qemu-devel@nongnu.org ; > > jonathan.came...@huawei.com ; > > linux-...@vger.kernel.org ; > > gregory.pr...@memverge.com ; > > hch...@avery-design.com.tw ; > > cbr...@avery-design.com ; dan.j.willi...@intel.com > > ; Adam Manzanares ; > > d...@stgolabs.net ; nmtadam.sams...@gmail.com > > ; ni...@outlook.com > > Subject: Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > > > On Mon, Jun 05, 2023 at 10:35:48AM -0700, Ira Weiny wrote: > > > Fan Ni wrote: > > > > Since the early draft of DCD support in kernel is out > > > > (https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21LzAGNOk$ > > > > ), > > > > this patch series provide dcd emulation in qemu so people who are > > > > interested > > > > can have an early try. It is noted that the patch series may need to be > > > > updated > > > > accordingly if the kernel side implementation changes. > > > > > > Fan, > > > > > > Do you have a git tree we can pull this from whic
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
Shesha Bhushan Sreenivasamurthy wrote: > Hi Fan, >I am implementing DCD FMAPI commands and planning to start pushing changes > to the below branch. That requires the contributions you have made. Can your > changes be pushed to the below branch ? > > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25 This is the branch I'm trying to use as well. Thanks, Ira
Re: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
From: Jonathan Cameron Sent: Thursday, June 8, 2023 2:43 AM To: Shesha Bhushan Sreenivasamurthy Cc: Fan Ni ; qemu-devel@nongnu.org ; linux-...@vger.kernel.org ; gregory.pr...@memverge.com ; hch...@avery-design.com.tw ; cbr...@avery-design.com ; dan.j.willi...@intel.com ; Adam Manzanares ; d...@stgolabs.net ; nmtadam.sams...@gmail.com ; ni...@outlook.com ; Ira Weiny Subject: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu External Email -- On Wed, 7 Jun 2023 18:52:14 + Shesha Bhushan Sreenivasamurthy wrote: > From: Fan Ni > Sent: Wednesday, June 7, 2023 11:31 AM > To: Shesha Bhushan Sreenivasamurthy > Cc: Jonathan Cameron ; qemu-devel@nongnu.org > ; linux-...@vger.kernel.org > ; gregory.pr...@memverge.com > ; hch...@avery-design.com.tw > ; cbr...@avery-design.com > ; dan.j.willi...@intel.com > ; Adam Manzanares ; > d...@stgolabs.net ; nmtadam.sams...@gmail.com > ; ni...@outlook.com ; Ira Weiny > > Subject: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > External Email > > -- > On Wed, Jun 07, 2023 at 06:13:01PM +, Shesha Bhushan Sreenivasamurthy > wrote: > > Hi Fan, > > I am implementing DCD FMAPI commands and planning to start pushing > >changes to the below branch. That requires the contributions you have made. > >Can your changes be pushed to the below branch ? > > > > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25__;!!EwVzqGoTKBqv-0DWAJBm!Vt5uIqwW-L4c4gh02ulI4M762JNQ3_aE9k9lb6QlwE2xm6T23ic7ig7Y77i1VN7l_RX_ySIQhre_z7Q0JA$ > > > > Can you push changes to the branch directly? I think it is Jonathan's private > branch. However, I can fork the branch and rebase my patch series atop and > share with you the new repo if that helps you move forward your > work. > Let me know your thought. > > ss - I saw commits from others, so assumed you can. Since it is Jonathan's > private repo, I will step back and let him answer. I tend to apply stuff in manually rather than given more people commit access to that particular gitlab tree. Easiest option is to fork on gitlab and share the path of your own fork. I normally queue reasonably mature stuff up on my tree, but that's about managing the series sent with intent of being applied upstream + providing a fairly stable test branch. It's not intended as a general place for stuff in development (though I might sneak out an extra branch myself from time to time if I want to talk about it :) Jonathan ss - Thanks ! I have forked Jonathan's repo on Gitlab and will work of cxl-2023-05-25 branch. I will apply Fan's patch manually and continue .. > > Fan > > > > > > > From: Fan Ni > > Sent: Monday, June 5, 2023 10:51 AM > > To: Ira Weiny > > Cc: qemu-devel@nongnu.org ; > > jonathan.came...@huawei.com ; > > linux-...@vger.kernel.org ; > > gregory.pr...@memverge.com ; > > hch...@avery-design.com.tw ; > > cbr...@avery-design.com ; dan.j.willi...@intel.com > > ; Adam Manzanares ; > > d...@stgolabs.net ; nmtadam.sams...@gmail.com > > ; ni...@outlook.com > > Subject: Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > > > On Mon, Jun 05, 2023 at 10:35:48AM -0700, Ira Weiny wrote: > > > Fan Ni wrote: > > > > Since the early draft of DCD support in kernel is out > > > > (https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21LzAGNOk$ > > > > ), > > > > this patch series provide dcd emulation in qemu so people who are > > > > interested > > > > can have an early try. It is noted that the patch series may need to be > > > > updated > > > > accordingly if the kernel side implementation changes. > > > > > > Fan, > > > > > > Do you have a git tree we can pull this from which is updated to a more > > > recent CXL branch from Jonathan? > > > > > > Thanks, > > > Ira > > > > Hi Ira, > > > > I have a git tree of the patch series based on Jonathan's branch > > cxl-2023-02-28: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_moking_qemu-2Ddev_tree_dcd-2Drfe=DwIFAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=w6dicn5kXEG4Imk6TpICIjdA6KJ-xt84dtHui-Y0fv5H13bijtzEvjxECKE5MHYf=3yeO9RN5FY3gPfO2y19X057YeqRTTQTQNfNA-Gfir_Q= > > . &g
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Wed, 7 Jun 2023 18:52:14 + Shesha Bhushan Sreenivasamurthy wrote: > From: Fan Ni > Sent: Wednesday, June 7, 2023 11:31 AM > To: Shesha Bhushan Sreenivasamurthy > Cc: Jonathan Cameron ; qemu-devel@nongnu.org > ; linux-...@vger.kernel.org > ; gregory.pr...@memverge.com > ; hch...@avery-design.com.tw > ; cbr...@avery-design.com > ; dan.j.willi...@intel.com > ; Adam Manzanares ; > d...@stgolabs.net ; nmtadam.sams...@gmail.com > ; ni...@outlook.com ; Ira Weiny > > Subject: [EXT] Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > External Email > > -- > On Wed, Jun 07, 2023 at 06:13:01PM +, Shesha Bhushan Sreenivasamurthy > wrote: > > Hi Fan, > > I am implementing DCD FMAPI commands and planning to start pushing > >changes to the below branch. That requires the contributions you have made. > >Can your changes be pushed to the below branch ? > > > > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25__;!!EwVzqGoTKBqv-0DWAJBm!Vt5uIqwW-L4c4gh02ulI4M762JNQ3_aE9k9lb6QlwE2xm6T23ic7ig7Y77i1VN7l_RX_ySIQhre_z7Q0JA$ > > > > Can you push changes to the branch directly? I think it is Jonathan's private > branch. However, I can fork the branch and rebase my patch series atop and > share with you the new repo if that helps you move forward your > work. > Let me know your thought. > > ss - I saw commits from others, so assumed you can. Since it is Jonathan's > private repo, I will step back and let him answer. I tend to apply stuff in manually rather than given more people commit access to that particular gitlab tree. Easiest option is to fork on gitlab and share the path of your own fork. I normally queue reasonably mature stuff up on my tree, but that's about managing the series sent with intent of being applied upstream + providing a fairly stable test branch. It's not intended as a general place for stuff in development (though I might sneak out an extra branch myself from time to time if I want to talk about it :) Jonathan > > Fan > > > > > > > From: Fan Ni > > Sent: Monday, June 5, 2023 10:51 AM > > To: Ira Weiny > > Cc: qemu-devel@nongnu.org ; > > jonathan.came...@huawei.com ; > > linux-...@vger.kernel.org ; > > gregory.pr...@memverge.com ; > > hch...@avery-design.com.tw ; > > cbr...@avery-design.com ; dan.j.willi...@intel.com > > ; Adam Manzanares ; > > d...@stgolabs.net ; nmtadam.sams...@gmail.com > > ; ni...@outlook.com > > Subject: Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > > > On Mon, Jun 05, 2023 at 10:35:48AM -0700, Ira Weiny wrote: > > > Fan Ni wrote: > > > > Since the early draft of DCD support in kernel is out > > > > (https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21LzAGNOk$ > > > > ), > > > > this patch series provide dcd emulation in qemu so people who are > > > > interested > > > > can have an early try. It is noted that the patch series may need to be > > > > updated > > > > accordingly if the kernel side implementation changes. > > > > > > Fan, > > > > > > Do you have a git tree we can pull this from which is updated to a more > > > recent CXL branch from Jonathan? > > > > > > Thanks, > > > Ira > > > > Hi Ira, > > > > I have a git tree of the patch series based on Jonathan's branch > > cxl-2023-02-28: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_moking_qemu-2Ddev_tree_dcd-2Drfe=DwIFAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=w6dicn5kXEG4Imk6TpICIjdA6KJ-xt84dtHui-Y0fv5H13bijtzEvjxECKE5MHYf=3yeO9RN5FY3gPfO2y19X057YeqRTTQTQNfNA-Gfir_Q= > > . > > > > That may be not new enough to include some of the recent patches, but I can > > rebase it to a newer branch if you can tell me which branch you want to use. > > > > Thanks, > > Fan > > > > > > > > > > > > > To support DCD emulation, the patch series add DCD related mailbox > > > > command > > > > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory > > > > device > > > > with dynamic capacity extent and region representative. > > > > To support read/write to the dynamic capacity of the device, a host > > > > backend &
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
Hi Fan,
I am implementing DCD FMAPI commands and planning to start pushing changes
to the below branch. That requires the contributions you have made. Can your
changes be pushed to the below branch ?
https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25
From: Fan Ni
Sent: Monday, June 5, 2023 10:51 AM
To: Ira Weiny
Cc: qemu-devel@nongnu.org ; jonathan.came...@huawei.com
; linux-...@vger.kernel.org
; gregory.pr...@memverge.com
; hch...@avery-design.com.tw
; cbr...@avery-design.com
; dan.j.willi...@intel.com ;
Adam Manzanares ; d...@stgolabs.net
; nmtadam.sams...@gmail.com ;
ni...@outlook.com
Subject: Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Mon, Jun 05, 2023 at 10:35:48AM -0700, Ira Weiny wrote:
> Fan Ni wrote:
> > Since the early draft of DCD support in kernel is out
> > (https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21LzAGNOk$
> > ),
> > this patch series provide dcd emulation in qemu so people who are interested
> > can have an early try. It is noted that the patch series may need to be
> > updated
> > accordingly if the kernel side implementation changes.
>
> Fan,
>
> Do you have a git tree we can pull this from which is updated to a more
> recent CXL branch from Jonathan?
>
> Thanks,
> Ira
Hi Ira,
I have a git tree of the patch series based on Jonathan's branch
cxl-2023-02-28:
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_moking_qemu-2Ddev_tree_dcd-2Drfe=DwIFAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=w6dicn5kXEG4Imk6TpICIjdA6KJ-xt84dtHui-Y0fv5H13bijtzEvjxECKE5MHYf=3yeO9RN5FY3gPfO2y19X057YeqRTTQTQNfNA-Gfir_Q=
.
That may be not new enough to include some of the recent patches, but I can
rebase it to a newer branch if you can tell me which branch you want to use.
Thanks,
Fan
>
> >
> > To support DCD emulation, the patch series add DCD related mailbox command
> > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device
> > with dynamic capacity extent and region representative.
> > To support read/write to the dynamic capacity of the device, a host backend
> > is provided and necessary check mechnism is added to ensure the dynamic
> > capacity accessed is backed with active dc extents.
> > Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not
> > supported
> > , but we add two qmp interfaces for adding/releasing dynamic capacity
> > extents.
> > Also, the support for multiple hosts sharing the same DCD case is missing.
> >
> > Things we can try with the patch series together with kernel dcd code:
> > 1. Create DC regions to cover the address range of the dynamic capacity
> > regions.
> > 2. Add/release dynamic capacity extents to the device and notify the
> > kernel.
> > 3. Test kernel side code to accept added dc extents and create dax devices,
> > and release dc extents and notify the device
> > 4. Online the memory range backed with dc extents and let application use
> > them.
> >
> > The patch series is based on Jonathan's local qemu branch:
> > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28__;!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21OO3UHEM$
> >
> >
> > Simple tests peformed with the patch series:
> > 1 Install cxl modules:
> >
> > modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem
> >
> > 2 Create dc regions:
> >
> > region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region)
> > echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region
> > echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity
> > echo 1 > /sys/bus/cxl/devices/$region/interleave_ways
> > echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode
> > echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size
> > echo 0x1000 > /sys/bus/cxl/devices/$region/size
> > echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0
> > echo 1 > /sys/bus/cxl/devices/$region/commit
> > echo $region > /sys/bus/cxl/drivers/cxl_region/bind
> >
> > /home/fan/cxl/tools-and-scripts# cxl list
> > [
> > {
> > "memdevs":[
> > {
> > "memdev":"mem0",
> > "pmem_size":536870912,
> > "ram_size":0,
> > "serial":0,
> > "host":":0d:00.0"
> > }
> >
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Wed, Jun 07, 2023 at 06:13:01PM +, Shesha Bhushan Sreenivasamurthy wrote: > Hi Fan, >I am implementing DCD FMAPI commands and planning to start pushing changes > to the below branch. That requires the contributions you have made. Can your > changes be pushed to the below branch ? > > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25__;!!EwVzqGoTKBqv-0DWAJBm!Vt5uIqwW-L4c4gh02ulI4M762JNQ3_aE9k9lb6QlwE2xm6T23ic7ig7Y77i1VN7l_RX_ySIQhre_z7Q0JA$ > Can you push changes to the branch directly? I think it is Jonathan's private branch. However, I can fork the branch and rebase my patch series atop and share with you the new repo if that helps you move forward your work. Let me know your thought. Fan > > > From: Fan Ni > Sent: Monday, June 5, 2023 10:51 AM > To: Ira Weiny > Cc: qemu-devel@nongnu.org ; > jonathan.came...@huawei.com ; > linux-...@vger.kernel.org ; > gregory.pr...@memverge.com ; > hch...@avery-design.com.tw ; > cbr...@avery-design.com ; dan.j.willi...@intel.com > ; Adam Manzanares ; > d...@stgolabs.net ; nmtadam.sams...@gmail.com > ; ni...@outlook.com > Subject: Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu > > On Mon, Jun 05, 2023 at 10:35:48AM -0700, Ira Weiny wrote: > > Fan Ni wrote: > > > Since the early draft of DCD support in kernel is out > > > (https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21LzAGNOk$ > > > ), > > > this patch series provide dcd emulation in qemu so people who are > > > interested > > > can have an early try. It is noted that the patch series may need to be > > > updated > > > accordingly if the kernel side implementation changes. > > > > Fan, > > > > Do you have a git tree we can pull this from which is updated to a more > > recent CXL branch from Jonathan? > > > > Thanks, > > Ira > > Hi Ira, > > I have a git tree of the patch series based on Jonathan's branch > cxl-2023-02-28: > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_moking_qemu-2Ddev_tree_dcd-2Drfe=DwIFAg=nKjWec2b6R0mOyPaz7xtfQ=Zta64bwn4nurTRpD4LY2OGr8KklkMRPn7Z_Qy0o4unU=w6dicn5kXEG4Imk6TpICIjdA6KJ-xt84dtHui-Y0fv5H13bijtzEvjxECKE5MHYf=3yeO9RN5FY3gPfO2y19X057YeqRTTQTQNfNA-Gfir_Q= > . > > That may be not new enough to include some of the recent patches, but I can > rebase it to a newer branch if you can tell me which branch you want to use. > > Thanks, > Fan > > > > > > > > > To support DCD emulation, the patch series add DCD related mailbox command > > > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device > > > with dynamic capacity extent and region representative. > > > To support read/write to the dynamic capacity of the device, a host > > > backend > > > is provided and necessary check mechnism is added to ensure the dynamic > > > capacity accessed is backed with active dc extents. > > > Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not > > > supported > > > , but we add two qmp interfaces for adding/releasing dynamic capacity > > > extents. > > > Also, the support for multiple hosts sharing the same DCD case is missing. > > > > > > Things we can try with the patch series together with kernel dcd code: > > > 1. Create DC regions to cover the address range of the dynamic capacity > > > regions. > > > 2. Add/release dynamic capacity extents to the device and notify the > > > kernel. > > > 3. Test kernel side code to accept added dc extents and create dax > > > devices, > > > and release dc extents and notify the device > > > 4. Online the memory range backed with dc extents and let application use > > > them. > > > > > > The patch series is based on Jonathan's local qemu branch: > > > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28__;!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21OO3UHEM$ > > > > > > > > > Simple tests peformed with the patch series: > > > 1 Install cxl modules: > > > > > > modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem > > > > > > 2 Create dc regions: > > > > > > region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) > > > echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region > > > ec
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Mon, Jun 05, 2023 at 10:35:48AM -0700, Ira Weiny wrote:
> Fan Ni wrote:
> > Since the early draft of DCD support in kernel is out
> > (https://urldefense.com/v3/__https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/*t__;Iw!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21LzAGNOk$
> > ),
> > this patch series provide dcd emulation in qemu so people who are interested
> > can have an early try. It is noted that the patch series may need to be
> > updated
> > accordingly if the kernel side implementation changes.
>
> Fan,
>
> Do you have a git tree we can pull this from which is updated to a more
> recent CXL branch from Jonathan?
>
> Thanks,
> Ira
Hi Ira,
I have a git tree of the patch series based on Jonathan's branch
cxl-2023-02-28: https://github.com/moking/qemu-dev/tree/dcd-rfe.
That may be not new enough to include some of the recent patches, but I can
rebase it to a newer branch if you can tell me which branch you want to use.
Thanks,
Fan
>
> >
> > To support DCD emulation, the patch series add DCD related mailbox command
> > support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device
> > with dynamic capacity extent and region representative.
> > To support read/write to the dynamic capacity of the device, a host backend
> > is provided and necessary check mechnism is added to ensure the dynamic
> > capacity accessed is backed with active dc extents.
> > Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not
> > supported
> > , but we add two qmp interfaces for adding/releasing dynamic capacity
> > extents.
> > Also, the support for multiple hosts sharing the same DCD case is missing.
> >
> > Things we can try with the patch series together with kernel dcd code:
> > 1. Create DC regions to cover the address range of the dynamic capacity
> > regions.
> > 2. Add/release dynamic capacity extents to the device and notify the
> > kernel.
> > 3. Test kernel side code to accept added dc extents and create dax devices,
> > and release dc extents and notify the device
> > 4. Online the memory range backed with dc extents and let application use
> > them.
> >
> > The patch series is based on Jonathan's local qemu branch:
> > https://urldefense.com/v3/__https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28__;!!EwVzqGoTKBqv-0DWAJBm!RHzXPIcSiGsqUciUIH6HnlG_W--4L5CHfvcOIeUFdwKFhAujXuFDxjymmpCdOu7SLr61rww7lr21OO3UHEM$
> >
> >
> > Simple tests peformed with the patch series:
> > 1 Install cxl modules:
> >
> > modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem
> >
> > 2 Create dc regions:
> >
> > region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region)
> > echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region
> > echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity
> > echo 1 > /sys/bus/cxl/devices/$region/interleave_ways
> > echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode
> > echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size
> > echo 0x1000 > /sys/bus/cxl/devices/$region/size
> > echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0
> > echo 1 > /sys/bus/cxl/devices/$region/commit
> > echo $region > /sys/bus/cxl/drivers/cxl_region/bind
> >
> > /home/fan/cxl/tools-and-scripts# cxl list
> > [
> > {
> > "memdevs":[
> > {
> > "memdev":"mem0",
> > "pmem_size":536870912,
> > "ram_size":0,
> > "serial":0,
> > "host":":0d:00.0"
> > }
> > ]
> > },
> > {
> > "regions":[
> > {
> > "region":"region0",
> > "resource":45365592064,
> > "size":268435456,
> > "interleave_ways":1,
> > "interleave_granularity":256,
> > "decode_state":"commit"
> > }
> > ]
> > }
> > ]
> >
> > 3 Add two dc extents (128MB each) through qmp interface
> >
> > { "execute": "qmp_capabilities" }
> >
> > { "execute": "cxl-add-dynamic-capacity-event",
> > "arguments": {
> > "path": "/machine/peripheral/cxl-pmem0",
> > "region-id" : 0,
> > "num-extent": 2,
> > "dpa":0,
> > "extent-len": 128
> > }
> > }
> >
> > /home/fan/cxl/tools-and-scripts# lsmem
> > RANGE SIZE STATE REMOVABLE BLOCK
> > 0x-0x7fff2G online yes0-15
> > 0x0001-0x00027fff6G online yes 32-79
> > 0x000a9000-0x000a9fff 256M offline 338-339
> >
> > Memory block size: 128M
> > Total online memory: 8G
> > Total offline memory:256M
> >
> >
> > 4.Online the momory with 'daxctl online-memory dax0.0' to online the memory
> >
> > /home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0
> > [ 230.730553] Fallback order for Node 0: 0 1
> > [ 230.730825] Fallback order for Node 1: 1 0
> > [ 230.730953] Built 2 zonelists, mobility grouping on. Total pages:
> >
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
Fan Ni wrote:
> Since the early draft of DCD support in kernel is out
> (https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/#t),
> this patch series provide dcd emulation in qemu so people who are interested
> can have an early try. It is noted that the patch series may need to be
> updated
> accordingly if the kernel side implementation changes.
Fan,
Do you have a git tree we can pull this from which is updated to a more
recent CXL branch from Jonathan?
Thanks,
Ira
>
> To support DCD emulation, the patch series add DCD related mailbox command
> support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device
> with dynamic capacity extent and region representative.
> To support read/write to the dynamic capacity of the device, a host backend
> is provided and necessary check mechnism is added to ensure the dynamic
> capacity accessed is backed with active dc extents.
> Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not supported
> , but we add two qmp interfaces for adding/releasing dynamic capacity extents.
> Also, the support for multiple hosts sharing the same DCD case is missing.
>
> Things we can try with the patch series together with kernel dcd code:
> 1. Create DC regions to cover the address range of the dynamic capacity
> regions.
> 2. Add/release dynamic capacity extents to the device and notify the
> kernel.
> 3. Test kernel side code to accept added dc extents and create dax devices,
> and release dc extents and notify the device
> 4. Online the memory range backed with dc extents and let application use
> them.
>
> The patch series is based on Jonathan's local qemu branch:
> https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28
>
> Simple tests peformed with the patch series:
> 1 Install cxl modules:
>
> modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem
>
> 2 Create dc regions:
>
> region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region)
> echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region
> echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity
> echo 1 > /sys/bus/cxl/devices/$region/interleave_ways
> echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode
> echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size
> echo 0x1000 > /sys/bus/cxl/devices/$region/size
> echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0
> echo 1 > /sys/bus/cxl/devices/$region/commit
> echo $region > /sys/bus/cxl/drivers/cxl_region/bind
>
> /home/fan/cxl/tools-and-scripts# cxl list
> [
> {
> "memdevs":[
> {
> "memdev":"mem0",
> "pmem_size":536870912,
> "ram_size":0,
> "serial":0,
> "host":":0d:00.0"
> }
> ]
> },
> {
> "regions":[
> {
> "region":"region0",
> "resource":45365592064,
> "size":268435456,
> "interleave_ways":1,
> "interleave_granularity":256,
> "decode_state":"commit"
> }
> ]
> }
> ]
>
> 3 Add two dc extents (128MB each) through qmp interface
>
> { "execute": "qmp_capabilities" }
>
> { "execute": "cxl-add-dynamic-capacity-event",
> "arguments": {
>"path": "/machine/peripheral/cxl-pmem0",
> "region-id" : 0,
>"num-extent": 2,
> "dpa":0,
> "extent-len": 128
> }
> }
>
> /home/fan/cxl/tools-and-scripts# lsmem
> RANGE SIZE STATE REMOVABLE BLOCK
> 0x-0x7fff2G online yes0-15
> 0x0001-0x00027fff6G online yes 32-79
> 0x000a9000-0x000a9fff 256M offline 338-339
>
> Memory block size: 128M
> Total online memory: 8G
> Total offline memory:256M
>
>
> 4.Online the momory with 'daxctl online-memory dax0.0' to online the memory
>
> /home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0
> [ 230.730553] Fallback order for Node 0: 0 1
> [ 230.730825] Fallback order for Node 1: 1 0
> [ 230.730953] Built 2 zonelists, mobility grouping on. Total pages: 2042541
> [ 230.731110] Policy zone: Normal
> onlined memory for 1 device
>
> root@bgt-140510-bm03:/home/fan/cxl/ndctl# lsmem
> RANGE SIZE STATE REMOVABLE BLOCK
> 0x-0x7fff2G online yes 0-15
> 0x0001-0x00027fff6G online yes 32-79
> 0x000a9000-0x000a97ff 128M online yes 338
> 0x000a9800-0x000a9fff 128M offline 339
>
> Memory block size: 128M
> Total online memory: 8.1G
> Total offline memory:128M
>
> 5 using dc extents as regular memory
>
> /home/fan/cxl/ndctl# numactl --membind=1 ls
> CONTRIBUTING.md README.md clean_config.sh cscope.out git-version-gen
> ndctlscripts test.h version.h.in COPYING acpi.h
> config.h.meson cxlmake-git-snapshot.sh ndctl.spec.in sles
RE: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
-Original Message-
From: Jonathan Cameron
Sent: Monday, May 15, 2023 6:31 PM
To: Fan Ni
Cc: qemu-devel@nongnu.org; linux-...@vger.kernel.org;
gregory.pr...@memverge.com; hch...@avery-design.com.tw; Browy, Christopher
; Weiny, Ira ; Williams, Dan J
; Adam Manzanares ;
d...@stgolabs.net; nmtadam.sams...@gmail.com; ni...@outlook.com; Singh, Navneet
Subject: Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Thu, 11 May 2023 17:56:40 +
Fan Ni wrote:
> Since the early draft of DCD support in kernel is out
> (https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510
> -bm03/T/#t), this patch series provide dcd emulation in qemu so people
> who are interested can have an early try. It is noted that the patch
> series may need to be updated accordingly if the kernel side
> implementation changes.
>
> To support DCD emulation, the patch series add DCD related mailbox
> command support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3
> memory device with dynamic capacity extent and region representative.
> To support read/write to the dynamic capacity of the device, a host
> backend is provided and necessary check mechnism is added to ensure
> the dynamic capacity accessed is backed with active dc extents.
> Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not
> supported , but we add two qmp interfaces for adding/releasing dynamic
> capacity extents.
> Also, the support for multiple hosts sharing the same DCD case is missing.
>
> Things we can try with the patch series together with kernel dcd code:
> 1. Create DC regions to cover the address range of the dynamic
> capacity regions.
> 2. Add/release dynamic capacity extents to the device and notify the
> kernel.
> 3. Test kernel side code to accept added dc extents and create dax
> devices, and release dc extents and notify the device 4. Online the
> memory range backed with dc extents and let application use them.
>
> The patch series is based on Jonathan's local qemu branch:
> https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28
>
> Simple tests peformed with the patch series:
> 1 Install cxl modules:
>
> modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem
>
> 2 Create dc regions:
>
> region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region)
> echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region
> echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity
> echo 1 > /sys/bus/cxl/devices/$region/interleave_ways
> echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode
> echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size
> echo 0x1000 > /sys/bus/cxl/devices/$region/size echo "decoder2.0"
> > /sys/bus/cxl/devices/$region/target0
> echo 1 > /sys/bus/cxl/devices/$region/commit
> echo $region > /sys/bus/cxl/drivers/cxl_region/bind
>
> /home/fan/cxl/tools-and-scripts# cxl list [
> {
> "memdevs":[
> {
> "memdev":"mem0",
> "pmem_size":536870912,
> "ram_size":0,
> "serial":0,
> "host":":0d:00.0"
> }
> ]
> },
> {
> "regions":[
> {
> "region":"region0",
> "resource":45365592064,
> "size":268435456,
> "interleave_ways":1,
> "interleave_granularity":256,
> "decode_state":"commit"
> }
> ]
> }
> ]
>
> 3 Add two dc extents (128MB each) through qmp interface
>
> { "execute": "qmp_capabilities" }
>
> { "execute": "cxl-add-dynamic-capacity-event",
> "arguments": {
>"path": "/machine/peripheral/cxl-pmem0",
> "region-id" : 0,
>"num-extent": 2,
> "dpa":0,
> "extent-len": 128
> }
> }
>
> /home/fan/cxl/tools-and-scripts# lsmem
> RANGE SIZE STATE REMOVABLE BLOCK
> 0x-0x7fff2G online yes0-15
> 0x0001-0x00027fff6G online yes 32-79
> 0x000a9000-0x000a9fff 256M offline 338-339
>
> Memory block size: 128M
> Total online memory: 8G
> Total offline memory:256M
>
>
> 4.Online the momory with 'daxctl online-memory dax0.0' to online the
> memory
>
> /home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0 [
> 230.730553] Fallback order for Node 0: 0 1 [ 230.73
Re: [Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
On Thu, 11 May 2023 17:56:40 +
Fan Ni wrote:
> Since the early draft of DCD support in kernel is out
> (https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/#t),
> this patch series provide dcd emulation in qemu so people who are interested
> can have an early try. It is noted that the patch series may need to be
> updated
> accordingly if the kernel side implementation changes.
>
> To support DCD emulation, the patch series add DCD related mailbox command
> support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device
> with dynamic capacity extent and region representative.
> To support read/write to the dynamic capacity of the device, a host backend
> is provided and necessary check mechnism is added to ensure the dynamic
> capacity accessed is backed with active dc extents.
> Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not supported
> , but we add two qmp interfaces for adding/releasing dynamic capacity extents.
> Also, the support for multiple hosts sharing the same DCD case is missing.
>
> Things we can try with the patch series together with kernel dcd code:
> 1. Create DC regions to cover the address range of the dynamic capacity
> regions.
> 2. Add/release dynamic capacity extents to the device and notify the
> kernel.
> 3. Test kernel side code to accept added dc extents and create dax devices,
> and release dc extents and notify the device
> 4. Online the memory range backed with dc extents and let application use
> them.
>
> The patch series is based on Jonathan's local qemu branch:
> https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28
>
> Simple tests peformed with the patch series:
> 1 Install cxl modules:
>
> modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem
>
> 2 Create dc regions:
>
> region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region)
> echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region
> echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity
> echo 1 > /sys/bus/cxl/devices/$region/interleave_ways
> echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode
> echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size
> echo 0x1000 > /sys/bus/cxl/devices/$region/size
> echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0
> echo 1 > /sys/bus/cxl/devices/$region/commit
> echo $region > /sys/bus/cxl/drivers/cxl_region/bind
>
> /home/fan/cxl/tools-and-scripts# cxl list
> [
> {
> "memdevs":[
> {
> "memdev":"mem0",
> "pmem_size":536870912,
> "ram_size":0,
> "serial":0,
> "host":":0d:00.0"
> }
> ]
> },
> {
> "regions":[
> {
> "region":"region0",
> "resource":45365592064,
> "size":268435456,
> "interleave_ways":1,
> "interleave_granularity":256,
> "decode_state":"commit"
> }
> ]
> }
> ]
>
> 3 Add two dc extents (128MB each) through qmp interface
>
> { "execute": "qmp_capabilities" }
>
> { "execute": "cxl-add-dynamic-capacity-event",
> "arguments": {
>"path": "/machine/peripheral/cxl-pmem0",
> "region-id" : 0,
>"num-extent": 2,
> "dpa":0,
> "extent-len": 128
> }
> }
>
> /home/fan/cxl/tools-and-scripts# lsmem
> RANGE SIZE STATE REMOVABLE BLOCK
> 0x-0x7fff2G online yes0-15
> 0x0001-0x00027fff6G online yes 32-79
> 0x000a9000-0x000a9fff 256M offline 338-339
>
> Memory block size: 128M
> Total online memory: 8G
> Total offline memory:256M
>
>
> 4.Online the momory with 'daxctl online-memory dax0.0' to online the memory
>
> /home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0
> [ 230.730553] Fallback order for Node 0: 0 1
> [ 230.730825] Fallback order for Node 1: 1 0
> [ 230.730953] Built 2 zonelists, mobility grouping on. Total pages: 2042541
> [ 230.731110] Policy zone: Normal
> onlined memory for 1 device
>
> root@bgt-140510-bm03:/home/fan/cxl/ndctl# lsmem
> RANGE SIZE STATE REMOVABLE BLOCK
> 0x-0x7fff2G online yes 0-15
> 0x0001-0x00027fff6G online yes 32-79
> 0x000a9000-0x000a97ff 128M online yes 338
> 0x000a9800-0x000a9fff 128M offline 339
>
> Memory block size: 128M
> Total online memory: 8.1G
> Total offline memory:128M
>
> 5 using dc extents as regular memory
>
> /home/fan/cxl/ndctl# numactl --membind=1 ls
> CONTRIBUTING.md README.md clean_config.sh cscope.out git-version-gen
> ndctlscripts test.h version.h.in COPYING acpi.h
> config.h.meson cxlmake-git-snapshot.sh ndctl.spec.in sles tools
> Documentation build contrib daxctl meson.build
>
[Qemu RFC 0/7] Early enabling of DCD emulation in Qemu
Since the early draft of DCD support in kernel is out
(https://lore.kernel.org/linux-cxl/20230417164126.GA1904906@bgt-140510-bm03/T/#t),
this patch series provide dcd emulation in qemu so people who are interested
can have an early try. It is noted that the patch series may need to be updated
accordingly if the kernel side implementation changes.
To support DCD emulation, the patch series add DCD related mailbox command
support (CXL Spec 3.0: 8.2.9.8.9), and extend the cxl type3 memory device
with dynamic capacity extent and region representative.
To support read/write to the dynamic capacity of the device, a host backend
is provided and necessary check mechnism is added to ensure the dynamic
capacity accessed is backed with active dc extents.
Currently FM related mailbox commands (cxl spec 3.0: 7.6.7.6) is not supported
, but we add two qmp interfaces for adding/releasing dynamic capacity extents.
Also, the support for multiple hosts sharing the same DCD case is missing.
Things we can try with the patch series together with kernel dcd code:
1. Create DC regions to cover the address range of the dynamic capacity
regions.
2. Add/release dynamic capacity extents to the device and notify the
kernel.
3. Test kernel side code to accept added dc extents and create dax devices,
and release dc extents and notify the device
4. Online the memory range backed with dc extents and let application use
them.
The patch series is based on Jonathan's local qemu branch:
https://gitlab.com/jic23/qemu/-/tree/cxl-2023-02-28
Simple tests peformed with the patch series:
1 Install cxl modules:
modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem
2 Create dc regions:
region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region)
echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region
echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity
echo 1 > /sys/bus/cxl/devices/$region/interleave_ways
echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode
echo 0x1000 >/sys/bus/cxl/devices/decoder2.0/dpa_size
echo 0x1000 > /sys/bus/cxl/devices/$region/size
echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0
echo 1 > /sys/bus/cxl/devices/$region/commit
echo $region > /sys/bus/cxl/drivers/cxl_region/bind
/home/fan/cxl/tools-and-scripts# cxl list
[
{
"memdevs":[
{
"memdev":"mem0",
"pmem_size":536870912,
"ram_size":0,
"serial":0,
"host":":0d:00.0"
}
]
},
{
"regions":[
{
"region":"region0",
"resource":45365592064,
"size":268435456,
"interleave_ways":1,
"interleave_granularity":256,
"decode_state":"commit"
}
]
}
]
3 Add two dc extents (128MB each) through qmp interface
{ "execute": "qmp_capabilities" }
{ "execute": "cxl-add-dynamic-capacity-event",
"arguments": {
"path": "/machine/peripheral/cxl-pmem0",
"region-id" : 0,
"num-extent": 2,
"dpa":0,
"extent-len": 128
}
}
/home/fan/cxl/tools-and-scripts# lsmem
RANGE SIZE STATE REMOVABLE BLOCK
0x-0x7fff2G online yes0-15
0x0001-0x00027fff6G online yes 32-79
0x000a9000-0x000a9fff 256M offline 338-339
Memory block size: 128M
Total online memory: 8G
Total offline memory:256M
4.Online the momory with 'daxctl online-memory dax0.0' to online the memory
/home/fan/cxl/ndctl# ./build/daxctl/daxctl online-memory dax0.0
[ 230.730553] Fallback order for Node 0: 0 1
[ 230.730825] Fallback order for Node 1: 1 0
[ 230.730953] Built 2 zonelists, mobility grouping on. Total pages: 2042541
[ 230.731110] Policy zone: Normal
onlined memory for 1 device
root@bgt-140510-bm03:/home/fan/cxl/ndctl# lsmem
RANGE SIZE STATE REMOVABLE BLOCK
0x-0x7fff2G online yes 0-15
0x0001-0x00027fff6G online yes 32-79
0x000a9000-0x000a97ff 128M online yes 338
0x000a9800-0x000a9fff 128M offline 339
Memory block size: 128M
Total online memory: 8.1G
Total offline memory:128M
5 using dc extents as regular memory
/home/fan/cxl/ndctl# numactl --membind=1 ls
CONTRIBUTING.md README.md clean_config.sh cscope.out git-version-gen
ndctl scripts test.h version.h.in COPYING acpi.h
config.h.meson cxl make-git-snapshot.sh ndctl.spec.in sles tools
Documentationbuild contrib daxctl meson.build
rhel
tagstopology.png LICENSESccan cscope.files
git-version meson_options.txt rpmbuild.shtest util
QEMU command line cxl configuration:
RP1="-object
memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=512M \
-object
