Re: [PATCH] drm/amdgpu: Check resize bar register when system uses large bar

2023-12-29 Thread Mario Limonciello

On 12/20/2023 19:58, Ma, Jun wrote:

Hi Christian,


On 12/20/2023 10:10 PM, Christian König wrote:

Am 19.12.23 um 06:58 schrieb Ma Jun:

Print a warnning message if the system can't access
the resize bar register when using large bar.


Well pretty clear NAK, we have embedded use cases where this would
trigger an incorrect warning.

What should that be good for in the first place?


Some customer platforms do not enable mmconfig for various reasons, such as
bios bug, and therefore cannot access the GPU extend configuration
space through mmio.

Therefore, when the system enters the d3cold state and resumes,
the amdgpu driver fails to resume because the extend configuration
space registers of GPU can't be restored. At this point, Usually we
only see some failure dmesg log printed by amdgpu driver, it is
difficult to find the root cause.

So I thought it would be helpful to print some warning messages at
the beginning to identify problems quickly.


This doesn't yet have review comments with the holidays but I think this 
is a scalable solution to that specific issue:


https://lore.kernel.org/linux-pci/20231215220343.22523-1-mario.limoncie...@amd.com/

Can you try on one of these affected systems and see that it helps?



Regards,
Ma Jun


Regards,
Christian.



Signed-off-by: Ma Jun 
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 10 +-
   1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 4b694696930e..e7aedb4bd66e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -1417,6 +1417,12 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, 
u32 wb)
__clear_bit(wb, adev->wb.used);
   }
   
+static inline void amdgpu_find_rb_register(struct amdgpu_device *adev)

+{
+   if (!pci_find_ext_capability(adev->pdev, PCI_EXT_CAP_ID_REBAR))
+   DRM_WARN("System can't access the resize bar register,please 
check!!\n");
+}
+
   /**
* amdgpu_device_resize_fb_bar - try to resize FB BAR
*
@@ -1444,8 +1450,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device 
*adev)
   
   	/* skip if the bios has already enabled large BAR */

if (adev->gmc.real_vram_size &&
-   (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+   (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size)) {
+   amdgpu_find_rb_register(adev);
return 0;
+   }
   
   	/* Check if the root BUS has 64bit memory resources */

root = adev->pdev->bus;






[PATCH 7/7] drm/amd/display: Fix power_helpers.c codestyle

2023-12-29 Thread Marcelo Mendes Spessoto Junior
Place define macro expression inside () in power_helpers.c file

Signed-off-by: Marcelo Mendes Spessoto Junior 
---
 drivers/gpu/drm/amd/display/modules/power/power_helpers.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/display/modules/power/power_helpers.c 
b/drivers/gpu/drm/amd/display/modules/power/power_helpers.c
index 1675314a3..4e8af166e 100644
--- a/drivers/gpu/drm/amd/display/modules/power/power_helpers.c
+++ b/drivers/gpu/drm/amd/display/modules/power/power_helpers.c
@@ -31,7 +31,7 @@
 
 #define DIV_ROUNDUP(a, b) (((a)+((b)/2))/(b))
 #define bswap16_based_on_endian(big_endian, value) \
-   (big_endian) ? cpu_to_be16(value) : cpu_to_le16(value)
+   ((big_endian) ? cpu_to_be16(value) : cpu_to_le16(value))
 
 /* Possible Min Reduction config from least aggressive to most aggressive
  *  01 2 3 4 5 6 7 8 9 1011   12
-- 
2.39.2



[PATCH 6/7] drm/amd/display: Fix hdcp_log.h codestyle

2023-12-29 Thread Marcelo Mendes Spessoto Junior
Place HDCP_EVENT_TRACE(hdcp, event) macro content inside do while loop
to avoid if-else issues in hdcp_log.h file

Signed-off-by: Marcelo Mendes Spessoto Junior 
---
 drivers/gpu/drm/amd/display/modules/hdcp/hdcp_log.h | 10 ++
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/display/modules/hdcp/hdcp_log.h 
b/drivers/gpu/drm/amd/display/modules/hdcp/hdcp_log.h
index c62df3bcc..55a2c5d4c 100644
--- a/drivers/gpu/drm/amd/display/modules/hdcp/hdcp_log.h
+++ b/drivers/gpu/drm/amd/display/modules/hdcp/hdcp_log.h
@@ -86,10 +86,12 @@
 #define HDCP_CPIRQ_TRACE(hdcp) \
HDCP_LOG_FSM(hdcp, "[Link %d] --> CPIRQ", hdcp->config.index)
 #define HDCP_EVENT_TRACE(hdcp, event) \
-   if (event == MOD_HDCP_EVENT_WATCHDOG_TIMEOUT) \
-   HDCP_TIMEOUT_TRACE(hdcp); \
-   else if (event == MOD_HDCP_EVENT_CPIRQ) \
-   HDCP_CPIRQ_TRACE(hdcp)
+   do { \
+   if (event == MOD_HDCP_EVENT_WATCHDOG_TIMEOUT) \
+   HDCP_TIMEOUT_TRACE(hdcp); \
+   else if (event == MOD_HDCP_EVENT_CPIRQ) \
+   HDCP_CPIRQ_TRACE(hdcp) \
+   } while (0)
 /* TODO: find some way to tell if logging is off to save time */
 #define HDCP_DDC_READ_TRACE(hdcp, msg_name, msg, msg_size) do { \
mod_hdcp_dump_binary_message(msg, msg_size, hdcp->buf, \
-- 
2.39.2



[PATCH 5/7] drm/amd/display: Fix hdcp2_execution.c codestyle

2023-12-29 Thread Marcelo Mendes Spessoto Junior
Remove braces for single statement if expressions and change comparison
order for hdcp2_execution.c file

Signed-off-by: Marcelo Mendes Spessoto Junior 
---
 drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_execution.c | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_execution.c 
b/drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_execution.c
index 91c22b96e..733f22bed 100644
--- a/drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_execution.c
+++ b/drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_execution.c
@@ -208,9 +208,8 @@ static inline uint8_t get_device_count(struct mod_hdcp 
*hdcp)
 static enum mod_hdcp_status check_device_count(struct mod_hdcp *hdcp)
 {
/* Avoid device count == 0 to do authentication */
-   if (0 == get_device_count(hdcp)) {
+   if (get_device_count(hdcp) == 0)
return MOD_HDCP_STATUS_HDCP1_DEVICE_COUNT_MISMATCH_FAILURE;
-   }
 
/* Some MST display may choose to report the internal panel as an HDCP 
RX.   */
/* To update this condition with 1(because the immediate repeater's 
internal */
@@ -689,9 +688,8 @@ static enum mod_hdcp_status validate_stream_ready(struct 
mod_hdcp *hdcp,
if (is_hdmi_dvi_sl_hdcp(hdcp)) {
if (!process_rxstatus(hdcp, event_ctx, input, ))
goto out;
-   if (event_ctx->rx_id_list_ready) {
+   if (event_ctx->rx_id_list_ready)
goto out;
-   }
}
if (is_hdmi_dvi_sl_hdcp(hdcp))
if (!mod_hdcp_execute_and_set(check_stream_ready_available,
-- 
2.39.2



[PATCH 4/7] drm/amd/display: Fix hdcp_psp.h codestyle

2023-12-29 Thread Marcelo Mendes Spessoto Junior
Fix identation inside enum and place expressions in define macros inside
() for hdcp_psp.h file

Signed-off-by: Marcelo Mendes Spessoto Junior 
---
 drivers/gpu/drm/amd/display/modules/hdcp/hdcp_psp.h | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/display/modules/hdcp/hdcp_psp.h 
b/drivers/gpu/drm/amd/display/modules/hdcp/hdcp_psp.h
index 5b71bc96b..7844ea916 100644
--- a/drivers/gpu/drm/amd/display/modules/hdcp/hdcp_psp.h
+++ b/drivers/gpu/drm/amd/display/modules/hdcp/hdcp_psp.h
@@ -98,9 +98,9 @@ enum ta_dtm_encoder_type {
  * This enum defines software value for dio_output_type
  */
 typedef enum {
-TA_DTM_DIO_OUTPUT_TYPE__INVALID,
-TA_DTM_DIO_OUTPUT_TYPE__DIRECT,
-TA_DTM_DIO_OUTPUT_TYPE__DPIA
+   TA_DTM_DIO_OUTPUT_TYPE__INVALID,
+   TA_DTM_DIO_OUTPUT_TYPE__DIRECT,
+   TA_DTM_DIO_OUTPUT_TYPE__DPIA
 } ta_dtm_dio_output_type;
 
 struct ta_dtm_topology_update_input_v3 {
@@ -237,11 +237,11 @@ enum ta_hdcp2_hdcp2_msg_id_max_size {
 #define TA_HDCP__HDCP1_KSV_LIST_MAX_ENTRIES 127
 #define TA_HDCP__HDCP1_V_PRIME_SIZE 20
 #define TA_HDCP__HDCP2_TX_BUF_MAX_SIZE 
\
-   TA_HDCP_HDCP2_MSG_ID_MAX_SIZE__AKE_NO_STORED_KM + 
TA_HDCP_HDCP2_MSG_ID_MAX_SIZE__AKE_STORED_KM + 6
+   (TA_HDCP_HDCP2_MSG_ID_MAX_SIZE__AKE_NO_STORED_KM + 
TA_HDCP_HDCP2_MSG_ID_MAX_SIZE__AKE_STORED_KM + 6)
 
 // 64 bits boundaries
 #define TA_HDCP__HDCP2_RX_BUF_MAX_SIZE 
\
-   TA_HDCP_HDCP2_MSG_ID_MAX_SIZE__AKE_SEND_CERT + 
TA_HDCP_HDCP2_MSG_ID_MAX_SIZE__AKE_RECEIVER_INFO + 4
+   (TA_HDCP_HDCP2_MSG_ID_MAX_SIZE__AKE_SEND_CERT + 
TA_HDCP_HDCP2_MSG_ID_MAX_SIZE__AKE_RECEIVER_INFO + 4)
 
 enum ta_hdcp_status {
TA_HDCP_STATUS__SUCCESS = 0x00,
-- 
2.39.2



[PATCH 3/7] drm/amd/display: Fix freesync.c codestyle

2023-12-29 Thread Marcelo Mendes Spessoto Junior
Remove braces for single statement if expression for freesync.c file

Signed-off-by: Marcelo Mendes Spessoto Junior 
---
 drivers/gpu/drm/amd/display/modules/freesync/freesync.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/display/modules/freesync/freesync.c 
b/drivers/gpu/drm/amd/display/modules/freesync/freesync.c
index 47296d155..3955b7e4b 100644
--- a/drivers/gpu/drm/amd/display/modules/freesync/freesync.c
+++ b/drivers/gpu/drm/amd/display/modules/freesync/freesync.c
@@ -81,6 +81,7 @@ struct mod_freesync *mod_freesync_create(struct dc *dc)
 void mod_freesync_destroy(struct mod_freesync *mod_freesync)
 {
struct core_freesync *core_freesync = NULL;
+
if (mod_freesync == NULL)
return;
core_freesync = MOD_FREESYNC_TO_CORE(mod_freesync);
@@ -278,9 +279,8 @@ static void apply_below_the_range(struct core_freesync 
*core_freesync,
}
} else if (last_render_time_in_us > (max_render_time_in_us + 
in_out_vrr->btr.margin_in_us / 2)) {
/* Enter Below the Range */
-   if (!in_out_vrr->btr.btr_active) {
+   if (!in_out_vrr->btr.btr_active)
in_out_vrr->btr.btr_active = true;
-   }
}
 
/* BTR set to "not active" so disengage */
-- 
2.39.2



[PATCH 2/7] drm/amd/display: Fix hdcp_psp.c codestyle

2023-12-29 Thread Marcelo Mendes Spessoto Junior
Fix identation for hdcp_psp.c file

Signed-off-by: Marcelo Mendes Spessoto Junior 
---
 drivers/gpu/drm/amd/display/modules/hdcp/hdcp_psp.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/display/modules/hdcp/hdcp_psp.c 
b/drivers/gpu/drm/amd/display/modules/hdcp/hdcp_psp.c
index ee67a35c2..8c137d7c0 100644
--- a/drivers/gpu/drm/amd/display/modules/hdcp/hdcp_psp.c
+++ b/drivers/gpu/drm/amd/display/modules/hdcp/hdcp_psp.c
@@ -443,7 +443,7 @@ enum mod_hdcp_status 
mod_hdcp_hdcp1_enable_dp_stream_encryption(struct mod_hdcp
for (i = 0; i < MAX_NUM_OF_DISPLAYS; i++) {
 
if (hdcp->displays[i].adjust.disable || hdcp->displays[i].state 
!= MOD_HDCP_DISPLAY_ACTIVE)
-   continue;
+   continue;
 
memset(hdcp_cmd, 0, sizeof(struct ta_hdcp_shared_memory));
 
@@ -926,7 +926,7 @@ enum mod_hdcp_status 
mod_hdcp_hdcp2_enable_dp_stream_encryption(struct mod_hdcp
 
for (i = 0; i < MAX_NUM_OF_DISPLAYS; i++) {
if (hdcp->displays[i].adjust.disable || hdcp->displays[i].state 
!= MOD_HDCP_DISPLAY_ACTIVE)
-   continue;
+   continue;
 

hdcp_cmd->in_msg.hdcp2_enable_dp_stream_encryption.display_handle = 
hdcp->displays[i].index;

hdcp_cmd->in_msg.hdcp2_enable_dp_stream_encryption.session_handle = 
hdcp->auth.id;
-- 
2.39.2



[PATCH 1/7] drm/amd/display: Fix hdcp1_execution.c codestyle

2023-12-29 Thread Marcelo Mendes Spessoto Junior
Remove braces from single statement if expression in hdcp1_execution.c
file

Signed-off-by: Marcelo Mendes Spessoto Junior 
---
 drivers/gpu/drm/amd/display/modules/hdcp/hdcp1_execution.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/display/modules/hdcp/hdcp1_execution.c 
b/drivers/gpu/drm/amd/display/modules/hdcp/hdcp1_execution.c
index 1ddb4f5ea..182e7532d 100644
--- a/drivers/gpu/drm/amd/display/modules/hdcp/hdcp1_execution.c
+++ b/drivers/gpu/drm/amd/display/modules/hdcp/hdcp1_execution.c
@@ -63,6 +63,7 @@ static inline enum mod_hdcp_status 
check_hdcp_capable_dp(struct mod_hdcp *hdcp)
 static inline enum mod_hdcp_status check_r0p_available_dp(struct mod_hdcp 
*hdcp)
 {
enum mod_hdcp_status status;
+
if (is_dp_hdcp(hdcp)) {
status = (hdcp->auth.msg.hdcp1.bstatus &
DP_BSTATUS_R0_PRIME_READY) ?
@@ -131,9 +132,8 @@ static inline uint8_t get_device_count(struct mod_hdcp 
*hdcp)
 static inline enum mod_hdcp_status check_device_count(struct mod_hdcp *hdcp)
 {
/* Avoid device count == 0 to do authentication */
-   if (0 == get_device_count(hdcp)) {
+   if (get_device_count(hdcp) == 0)
return MOD_HDCP_STATUS_HDCP1_DEVICE_COUNT_MISMATCH_FAILURE;
-   }
 
/* Some MST display may choose to report the internal panel as an HDCP 
RX.
 * To update this condition with 1(because the immediate repeater's 
internal
-- 
2.39.2



[PATCH 0/7] drm/amd/display: Fix codestyle issues for modules

2023-12-29 Thread Marcelo Mendes Spessoto Junior
This patchset aims to fix most of codestyle issues present in the
modules directory

Marcelo Mendes Spessoto Junior (7):
  Fix hdcp1_execution.c codestyle
  Fix hdcp_psp.c codestyle
  Fix freesync.c codestyle
  Fix hdcp_psp.h codestyle
  Fix hdcp2_execution.c codestyle
  Fix hdcp_log.h codestyle
  Fix power_helpers.c codestyle

 .../gpu/drm/amd/display/modules/freesync/freesync.c|  4 ++--
 .../gpu/drm/amd/display/modules/hdcp/hdcp1_execution.c |  4 ++--
 .../gpu/drm/amd/display/modules/hdcp/hdcp2_execution.c |  6 ++
 drivers/gpu/drm/amd/display/modules/hdcp/hdcp_log.h| 10 ++
 drivers/gpu/drm/amd/display/modules/hdcp/hdcp_psp.c|  4 ++--
 drivers/gpu/drm/amd/display/modules/hdcp/hdcp_psp.h| 10 +-
 .../gpu/drm/amd/display/modules/power/power_helpers.c  |  2 +-
 7 files changed, 20 insertions(+), 20 deletions(-)

-- 
2.39.2



[RFC PATCH] drm/amd/display: fix bandwidth validation failure on DCN 2.1

2023-12-29 Thread Melissa Wen
IGT `amdgpu/amd_color/crtc-lut-accuracy` fails right at the beginning of
the test execution, during atomic check, because DC rejects the
bandwidth state for a fb sizing 64x64. The test was previously working
with the deprecated dc_commit_state(). Now using
dc_validate_with_context() approach, the atomic check needs to perform a
full state validation. Therefore, set fast_validation to false in the
dc_validate_global_state call for atomic check.

Fixes: b8272241ff9d ("drm/amd/display: Drop dc_commit_state in favor of 
dc_commit_streams")
Signed-off-by: Melissa Wen 
---

Hi,

It's a long story. I was inspecting this bug report:
- https://gitlab.freedesktop.org/drm/amd/-/issues/2016
and noticed the IGT test `igt@amdgpu/amd_color@crtc-lut-accuracy`
mentioned there wasn't even being executed on a laptop with DCN 2.1
(HP HP ENVY x360 Convertible 13-ay1xxx/8929). The test fails right at
the beginning due to an atomic check rejection, as below:

Starting subtest: crtc-lut-accuracy
(amd_color:14772) igt_kms-CRITICAL: Test assertion failure function 
igt_display_commit_atomic, file ../lib/igt_kms.c:4530:
(amd_color:14772) igt_kms-CRITICAL: Failed assertion: ret == 0
(amd_color:14772) igt_kms-CRITICAL: Last errno: 22, Invalid argument
(amd_color:14772) igt_kms-CRITICAL: error: -22 != 0
Stack trace:
  #0 ../lib/igt_core.c:1989 __igt_fail_assert()
  #1 [igt_display_commit_atomic+0x44]
  #2 ../tests/amdgpu/amd_color.c:159 __igt_uniquereal_main395()
  #3 ../tests/amdgpu/amd_color.c:395 main()
  #4 ../sysdeps/nptl/libc_start_call_main.h:74 __libc_start_call_main()
  #5 ../csu/libc-start.c:128 __libc_start_main@@GLIBC_2.34()
  #6 [_start+0x21]
Subtest crtc-lut-accuracy failed.

Checking dmesg, we can see that a bandwidth validation failure causes
the atomic check rejection:

[  711.147663] amdgpu :04:00.0: [drm] Mode Validation Warning: Unknown 
Status failed validation.
[  711.147667] [drm:amdgpu_dm_atomic_check [amdgpu]] DC global validation 
failure: Bandwidth validation failure (BW and Watermark) (13)
[  711.147772] [drm:amdgpu_irq_dispatch [amdgpu]] Unregistered interrupt 
src_id: 243 of client_id:8
[  711.148033] [drm:amdgpu_dm_atomic_check [amdgpu]] Atomic check failed with 
err: -22

I also noticed that the atomic check doesn't fail if I change the fb
width and height used in the test from 64 to 66, and I can get the test
execution back (and with success). However, I recall that all test cases
of IGT `amd_color` were working in the past, so I bisected and found the
first bad commit:

b8272241ff9d drm/amd/display: Drop dc_commit_state in favor of dc_commit_streams

Bringing the `dc_commit_state` machinery back also prevents the
bandwidth validation failure, but the commit above says
dc_commit_streams validation is more complete than dc_commit_state, so I
discarded this approach.

After some debugging and code inspection, I found out that avoiding fast
validation on dc_validate_global_state during atomic check solves the
issue, but I'm not sure if this change may affect performance. I
compared exec time of some IGT tests and didn't see any differences, but
I recognize it doesn't provide enough evidence.

What do you think about this change? Should I examine other things? Do
you see any potential issue that I should investigate? Could you
recommend a better approach to assess any side-effect of not enabling
fast validation in the atomic check?

Please, let me know your thoughts.

Happy New Year!

Melissa

 drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c 
b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
index 2845c884398e..4f51a7ad7a3c 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
@@ -10745,7 +10745,7 @@ static int amdgpu_dm_atomic_check(struct drm_device 
*dev,
DRM_DEBUG_DRIVER("drm_dp_mst_atomic_check() failed\n");
goto fail;
}
-   status = dc_validate_global_state(dc, dm_state->context, true);
+   status = dc_validate_global_state(dc, dm_state->context, false);
if (status != DC_OK) {
DRM_DEBUG_DRIVER("DC global validation failure: %s 
(%d)",
   dc_status_to_str(status), status);
-- 
2.43.0



[PATCH] drm/amdgpu: Drop 'fence' check in 'to_amdgpu_amdkfd_fence()'

2023-12-29 Thread Srinivasan Shanmugam
Return value of container_of(...) can't be null, so null check is not
required for 'fence'. Hence drop its NULL check.

Fixes the below:
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c:93 to_amdgpu_amdkfd_fence() 
warn: can 'fence' even be NULL?

Cc: Felix Kuehling 
Cc: Christian König 
Cc: Alex Deucher 
Signed-off-by: Srinivasan Shanmugam 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c
index 469785d33791..1ef758ac5076 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c
@@ -90,7 +90,7 @@ struct amdgpu_amdkfd_fence *to_amdgpu_amdkfd_fence(struct 
dma_fence *f)
return NULL;
 
fence = container_of(f, struct amdgpu_amdkfd_fence, base);
-   if (fence && f->ops == _fence_ops)
+   if (f->ops == _fence_ops)
return fence;
 
return NULL;
-- 
2.34.1



[PATCH v3] drm/amdkfd: Prefer kernel data types u8, u16, u32, u64 in amdkfd/kfd_priv.h

2023-12-29 Thread Srinivasan Shanmugam
Fix the following checks reported by checkpatch:

CHECK: Prefer kernel type 'u8' over 'uint8_t'
CHECK: Prefer kernel type 'u16' over 'uint16_t'
CHECK: Prefer kernel type 'u64' over 'uint64_t'
CHECK: Prefer kernel type 'u32' over 'uint32_t'

Cc: Felix Kuehling 
Cc: Christian König 
Cc: Alex Deucher 
Signed-off-by: Srinivasan Shanmugam 
---
v3:
  - updated u32, u16, u64 for missed variables in v2

 drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 448 +-
 1 file changed, 226 insertions(+), 222 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h 
b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
index 45366b4ca976..c161b5220fd7 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
@@ -74,7 +74,7 @@
 #define KFD_MMAP_GPU_ID_SHIFT 46
 #define KFD_MMAP_GPU_ID_MASK (((1ULL << KFD_GPU_ID_HASH_WIDTH) - 1) \
<< KFD_MMAP_GPU_ID_SHIFT)
-#define KFD_MMAP_GPU_ID(gpu_id) uint64_t)gpu_id) << KFD_MMAP_GPU_ID_SHIFT)\
+#define KFD_MMAP_GPU_ID(gpu_id) u64)gpu_id) << KFD_MMAP_GPU_ID_SHIFT)\
& KFD_MMAP_GPU_ID_MASK)
 #define KFD_MMAP_GET_GPU_ID(offset)((offset & KFD_MMAP_GPU_ID_MASK) \
>> KFD_MMAP_GPU_ID_SHIFT)
@@ -91,7 +91,7 @@
 
 /* Macro for allocating structures */
 #define kfd_alloc_struct(ptr_to_struct)\
-   ((typeof(ptr_to_struct)) kzalloc(sizeof(*ptr_to_struct), GFP_KERNEL))
+   ((typeof(ptr_to_struct))kzalloc(sizeof(*ptr_to_struct), GFP_KERNEL))
 
 #define KFD_MAX_NUM_OF_PROCESSES 512
 #define KFD_MAX_NUM_OF_QUEUES_PER_PROCESS 1024
@@ -145,13 +145,13 @@ enum kfd_ioctl_flags {
 */
KFD_IOC_FLAG_CHECKPOINT_RESTORE = BIT(0),
 };
+
 /*
  * Kernel module parameter to specify maximum number of supported queues per
  * device
  */
 extern int max_num_of_queues_per_device;
 
-
 /* Kernel module parameter to specify the scheduling policy */
 extern int sched_policy;
 
@@ -212,24 +212,24 @@ struct kfd_node;
 
 struct kfd_event_interrupt_class {
bool (*interrupt_isr)(struct kfd_node *dev,
-   const uint32_t *ih_ring_entry, uint32_t *patched_ihre,
+ const u32 *ih_ring_entry, u32 *patched_ihre,
bool *patched_flag);
void (*interrupt_wq)(struct kfd_node *dev,
-   const uint32_t *ih_ring_entry);
+const u32 *ih_ring_entry);
 };
 
 struct kfd_device_info {
-   uint32_t gfx_target_version;
+   u32 gfx_target_version;
const struct kfd_event_interrupt_class *event_interrupt_class;
unsigned int max_pasid_bits;
unsigned int max_no_of_hqd;
unsigned int doorbell_size;
size_t ih_ring_entry_size;
-   uint8_t num_of_watch_points;
-   uint16_t mqd_size_aligned;
+   u8 num_of_watch_points;
+   u16 mqd_size_aligned;
bool supports_cwsr;
bool needs_pci_atomics;
-   uint32_t no_atomic_fw_version;
+   u32 no_atomic_fw_version;
unsigned int num_sdma_queues_per_engine;
unsigned int num_reserved_sdma_queues_per_engine;
DECLARE_BITMAP(reserved_sdma_queues_bitmap, KFD_MAX_SDMA_QUEUES);
@@ -239,17 +239,17 @@ unsigned int kfd_get_num_sdma_engines(struct kfd_node 
*kdev);
 unsigned int kfd_get_num_xgmi_sdma_engines(struct kfd_node *kdev);
 
 struct kfd_mem_obj {
-   uint32_t range_start;
-   uint32_t range_end;
-   uint64_t gpu_addr;
-   uint32_t *cpu_ptr;
+   u32 range_start;
+   u32 range_end;
+   u64 gpu_addr;
+   u32 *cpu_ptr;
void *gtt_mem;
 };
 
 struct kfd_vmid_info {
-   uint32_t first_vmid_kfd;
-   uint32_t last_vmid_kfd;
-   uint32_t vmid_num_kfd;
+   u32 first_vmid_kfd;
+   u32 last_vmid_kfd;
+   u32 vmid_num_kfd;
 };
 
 #define MAX_KFD_NODES  8
@@ -267,7 +267,7 @@ struct kfd_node {
  */
struct kfd_vmid_info vm_info;
unsigned int id;/* topology stub index */
-   uint32_t xcc_mask; /* Instance mask of XCCs present */
+   u32 xcc_mask; /* Instance mask of XCCs present */
struct amdgpu_xcp *xcp;
 
/* Interrupts */
@@ -281,7 +281,7 @@ struct kfd_node {
 * from the HW ring into a SW ring.
 */
bool interrupts_active;
-   uint32_t interrupt_bitmap; /* Only used for GFX 9.4.3 */
+   u32 interrupt_bitmap; /* Only used for GFX 9.4.3 */
 
/* QCM Device instance */
struct device_queue_manager *dqm;
@@ -293,7 +293,7 @@ struct kfd_node {
/* Clients watching SMI events */
struct list_head smi_clients;
spinlock_t smi_lock;
-   uint32_t reset_seq_num;
+   u32 reset_seq_num;
 
/* SRAM ECC flag */
atomic_t sram_ecc_flag;
@@ -326,7 +326,7 @@ struct kfd_dev {
struct mutex doorbell_mutex;
 
void *gtt_mem;
-   uint64_t gtt_start_gpu_addr;
+   u64 

[PATCH v2] drm/amdkfd: Prefer kernel data types u8, u16, u32, u64 in amdkfd/kfd_priv.h

2023-12-29 Thread Srinivasan Shanmugam
Fix the following checks reported by checkpatch:

CHECK: Prefer kernel type 'u8' over 'uint8_t'
CHECK: Prefer kernel type 'u16' over 'uint16_t'
CHECK: Prefer kernel type 'u64' over 'uint64_t'
CHECK: Prefer kernel type 'u32' over 'uint32_t'

Cc: Felix Kuehling 
Cc: Christian König 
Cc: Alex Deucher 
Signed-off-by: Srinivasan Shanmugam 
---
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 368 +-
 1 file changed, 186 insertions(+), 182 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h 
b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
index 745024b31340..d492e1315f7a 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
@@ -91,7 +91,7 @@
 
 /* Macro for allocating structures */
 #define kfd_alloc_struct(ptr_to_struct)\
-   ((typeof(ptr_to_struct)) kzalloc(sizeof(*ptr_to_struct), GFP_KERNEL))
+   ((typeof(ptr_to_struct))kzalloc(sizeof(*ptr_to_struct), GFP_KERNEL))
 
 #define KFD_MAX_NUM_OF_PROCESSES 512
 #define KFD_MAX_NUM_OF_QUEUES_PER_PROCESS 1024
@@ -145,13 +145,13 @@ enum kfd_ioctl_flags {
 */
KFD_IOC_FLAG_CHECKPOINT_RESTORE = BIT(0),
 };
+
 /*
  * Kernel module parameter to specify maximum number of supported queues per
  * device
  */
 extern int max_num_of_queues_per_device;
 
-
 /* Kernel module parameter to specify the scheduling policy */
 extern int sched_policy;
 
@@ -212,24 +212,24 @@ struct kfd_node;
 
 struct kfd_event_interrupt_class {
bool (*interrupt_isr)(struct kfd_node *dev,
-   const uint32_t *ih_ring_entry, uint32_t *patched_ihre,
+ const u32 *ih_ring_entry, uint32_t *patched_ihre,
bool *patched_flag);
void (*interrupt_wq)(struct kfd_node *dev,
-   const uint32_t *ih_ring_entry);
+const u32 *ih_ring_entry);
 };
 
 struct kfd_device_info {
-   uint32_t gfx_target_version;
+   u32 gfx_target_version;
const struct kfd_event_interrupt_class *event_interrupt_class;
unsigned int max_pasid_bits;
unsigned int max_no_of_hqd;
unsigned int doorbell_size;
size_t ih_ring_entry_size;
-   uint8_t num_of_watch_points;
-   uint16_t mqd_size_aligned;
+   u8 num_of_watch_points;
+   u16 mqd_size_aligned;
bool supports_cwsr;
bool needs_pci_atomics;
-   uint32_t no_atomic_fw_version;
+   u32 no_atomic_fw_version;
unsigned int num_sdma_queues_per_engine;
unsigned int num_reserved_sdma_queues_per_engine;
DECLARE_BITMAP(reserved_sdma_queues_bitmap, KFD_MAX_SDMA_QUEUES);
@@ -239,17 +239,17 @@ unsigned int kfd_get_num_sdma_engines(struct kfd_node 
*kdev);
 unsigned int kfd_get_num_xgmi_sdma_engines(struct kfd_node *kdev);
 
 struct kfd_mem_obj {
-   uint32_t range_start;
-   uint32_t range_end;
-   uint64_t gpu_addr;
-   uint32_t *cpu_ptr;
+   u32 range_start;
+   u32 range_end;
+   u64 gpu_addr;
+   u32 *cpu_ptr;
void *gtt_mem;
 };
 
 struct kfd_vmid_info {
-   uint32_t first_vmid_kfd;
-   uint32_t last_vmid_kfd;
-   uint32_t vmid_num_kfd;
+   u32 first_vmid_kfd;
+   u32 last_vmid_kfd;
+   u32 vmid_num_kfd;
 };
 
 #define MAX_KFD_NODES  8
@@ -267,7 +267,7 @@ struct kfd_node {
  */
struct kfd_vmid_info vm_info;
unsigned int id;/* topology stub index */
-   uint32_t xcc_mask; /* Instance mask of XCCs present */
+   u32 xcc_mask; /* Instance mask of XCCs present */
struct amdgpu_xcp *xcp;
 
/* Interrupts */
@@ -281,7 +281,7 @@ struct kfd_node {
 * from the HW ring into a SW ring.
 */
bool interrupts_active;
-   uint32_t interrupt_bitmap; /* Only used for GFX 9.4.3 */
+   u32 interrupt_bitmap; /* Only used for GFX 9.4.3 */
 
/* QCM Device instance */
struct device_queue_manager *dqm;
@@ -293,7 +293,7 @@ struct kfd_node {
/* Clients watching SMI events */
struct list_head smi_clients;
spinlock_t smi_lock;
-   uint32_t reset_seq_num;
+   u32 reset_seq_num;
 
/* SRAM ECC flag */
atomic_t sram_ecc_flag;
@@ -326,7 +326,7 @@ struct kfd_dev {
struct mutex doorbell_mutex;
 
void *gtt_mem;
-   uint64_t gtt_start_gpu_addr;
+   u64 gtt_start_gpu_addr;
void *gtt_start_cpu_ptr;
void *gtt_sa_bitmap;
struct mutex gtt_sa_lock;
@@ -336,9 +336,9 @@ struct kfd_dev {
bool init_complete;
 
/* Firmware versions */
-   uint16_t mec_fw_version;
-   uint16_t mec2_fw_version;
-   uint16_t sdma_fw_version;
+   u16 mec_fw_version;
+   u16 mec2_fw_version;
+   u16 sdma_fw_version;
 
/* CWSR */
bool cwsr_enabled;
@@ -346,7 +346,7 @@ struct kfd_dev {
unsigned int cwsr_isa_size;
 
/* xGMI */
-   uint64_t hive_id;
+   u64 hive_id;
 

Re: [PATCH] drm/amdkfd: Fix type of 'dbg_flags' in 'struct kfd_process'

2023-12-29 Thread SRINIVASAN SHANMUGAM



On 12/21/2023 11:30 PM, Felix Kuehling wrote:

On 2023-12-21 12:39, Srinivasan Shanmugam wrote:

dbg_flags looks to be defined with incorrect data type; to process
multiple debug flag options, and hence defined dbg_flags as u32.

Fixes the below:

drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_packet_manager_v9.c:117 
pm_map_process_aldebaran() warn: maybe use && instead of &


Please add a Fixes-tag:

Fixes: 0de4ec9a0353 ("drm/amdgpu: prepare map process for 
multi-process debug devices")





Suggested-by: Lijo Lazar 
Cc: Felix Kuehling 
Cc: Christian König 
Cc: Alex Deucher 
Signed-off-by: Srinivasan Shanmugam 
---
  drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h 
b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h

index 45366b4ca976..745024b31340 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
@@ -970,7 +970,7 @@ struct kfd_process {
  struct work_struct debug_event_workarea;
    /* Tracks debug per-vmid request for debug flags */
-    bool dbg_flags;
+    u32 dbg_flags;


For consistency with the rest of this header file, I'd prefer we use 
uint32_t here. Other than that, the patch is


Thanks for reviews & feedbacks Felix! I have proposed an another patch 
"[PATCH] drm/amdkfd: Prefer kernel data types u8, u16, u32, u64 in 
amdkfd/kfd_priv.h", where I have changed to u8, u16, u32, u64, for 
consistency, which is the preferred way as per kernel coding style.


Regards,

Srini



Reviewed-by: Felix Kuehling 



    atomic_t poison;
  /* Queues are in paused stated because we are in the process of 
doing a CRIU checkpoint */


[PATCH] drm/amdgpu: Remove unreachable code in 'atom_skip_src_int()'

2023-12-29 Thread Srinivasan Shanmugam
Fixes the below:
drivers/gpu/drm/amd/amdgpu/atom.c:398 atom_skip_src_int() warn: ignoring 
unreachable code.

Cc: Christian König 
Cc: Alex Deucher 
Signed-off-by: Srinivasan Shanmugam 
---
 drivers/gpu/drm/amd/amdgpu/atom.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/atom.c 
b/drivers/gpu/drm/amd/amdgpu/atom.c
index 2c221000782c..a33e890c70d9 100644
--- a/drivers/gpu/drm/amd/amdgpu/atom.c
+++ b/drivers/gpu/drm/amd/amdgpu/atom.c
@@ -395,7 +395,6 @@ static void atom_skip_src_int(atom_exec_context *ctx, 
uint8_t attr, int *ptr)
(*ptr)++;
return;
}
-   return;
}
 }
 
-- 
2.34.1



[PATCH] drm/amdkfd: Prefer kernel data types u8, u16, u32, u64 in amdkfd/kfd_priv.h

2023-12-29 Thread Srinivasan Shanmugam
Fix the following checks reported by checkpatch:

CHECK: Prefer kernel type 'u8' over 'uint8_t'
CHECK: Prefer kernel type 'u16' over 'uint16_t'
CHECK: Prefer kernel type 'u64' over 'uint64_t'
CHECK: Prefer kernel type 'u32' over 'uint32_t'

Cc: Felix Kuehling 
Cc: Christian König 
Cc: Alex Deucher 
Signed-off-by: Srinivasan Shanmugam 
---
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 368 +-
 1 file changed, 186 insertions(+), 182 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h 
b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
index 745024b31340..9830f137f116 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
@@ -91,7 +91,7 @@
 
 /* Macro for allocating structures */
 #define kfd_alloc_struct(ptr_to_struct)\
-   ((typeof(ptr_to_struct)) kzalloc(sizeof(*ptr_to_struct), GFP_KERNEL))
+   ((typeof(ptr_to_struct))kzalloc(sizeof(*ptr_to_struct), GFP_KERNEL))
 
 #define KFD_MAX_NUM_OF_PROCESSES 512
 #define KFD_MAX_NUM_OF_QUEUES_PER_PROCESS 1024
@@ -145,13 +145,13 @@ enum kfd_ioctl_flags {
 */
KFD_IOC_FLAG_CHECKPOINT_RESTORE = BIT(0),
 };
+
 /*
  * Kernel module parameter to specify maximum number of supported queues per
  * device
  */
 extern int max_num_of_queues_per_device;
 
-
 /* Kernel module parameter to specify the scheduling policy */
 extern int sched_policy;
 
@@ -212,24 +212,24 @@ struct kfd_node;
 
 struct kfd_event_interrupt_class {
bool (*interrupt_isr)(struct kfd_node *dev,
-   const uint32_t *ih_ring_entry, uint32_t *patched_ihre,
+ const u32 *ih_ring_entry, uint32_t *patched_ihre,
bool *patched_flag);
void (*interrupt_wq)(struct kfd_node *dev,
-   const uint32_t *ih_ring_entry);
+const uint32_t *ih_ring_entry);
 };
 
 struct kfd_device_info {
-   uint32_t gfx_target_version;
+   u32 gfx_target_version;
const struct kfd_event_interrupt_class *event_interrupt_class;
unsigned int max_pasid_bits;
unsigned int max_no_of_hqd;
unsigned int doorbell_size;
size_t ih_ring_entry_size;
-   uint8_t num_of_watch_points;
-   uint16_t mqd_size_aligned;
+   u8 num_of_watch_points;
+   u16 mqd_size_aligned;
bool supports_cwsr;
bool needs_pci_atomics;
-   uint32_t no_atomic_fw_version;
+   u32 no_atomic_fw_version;
unsigned int num_sdma_queues_per_engine;
unsigned int num_reserved_sdma_queues_per_engine;
DECLARE_BITMAP(reserved_sdma_queues_bitmap, KFD_MAX_SDMA_QUEUES);
@@ -239,17 +239,17 @@ unsigned int kfd_get_num_sdma_engines(struct kfd_node 
*kdev);
 unsigned int kfd_get_num_xgmi_sdma_engines(struct kfd_node *kdev);
 
 struct kfd_mem_obj {
-   uint32_t range_start;
-   uint32_t range_end;
-   uint64_t gpu_addr;
-   uint32_t *cpu_ptr;
+   u32 range_start;
+   u32 range_end;
+   u64 gpu_addr;
+   u32 *cpu_ptr;
void *gtt_mem;
 };
 
 struct kfd_vmid_info {
-   uint32_t first_vmid_kfd;
-   uint32_t last_vmid_kfd;
-   uint32_t vmid_num_kfd;
+   u32 first_vmid_kfd;
+   u32 last_vmid_kfd;
+   u32 vmid_num_kfd;
 };
 
 #define MAX_KFD_NODES  8
@@ -267,7 +267,7 @@ struct kfd_node {
  */
struct kfd_vmid_info vm_info;
unsigned int id;/* topology stub index */
-   uint32_t xcc_mask; /* Instance mask of XCCs present */
+   u32 xcc_mask; /* Instance mask of XCCs present */
struct amdgpu_xcp *xcp;
 
/* Interrupts */
@@ -281,7 +281,7 @@ struct kfd_node {
 * from the HW ring into a SW ring.
 */
bool interrupts_active;
-   uint32_t interrupt_bitmap; /* Only used for GFX 9.4.3 */
+   u32 interrupt_bitmap; /* Only used for GFX 9.4.3 */
 
/* QCM Device instance */
struct device_queue_manager *dqm;
@@ -293,7 +293,7 @@ struct kfd_node {
/* Clients watching SMI events */
struct list_head smi_clients;
spinlock_t smi_lock;
-   uint32_t reset_seq_num;
+   u32 reset_seq_num;
 
/* SRAM ECC flag */
atomic_t sram_ecc_flag;
@@ -326,7 +326,7 @@ struct kfd_dev {
struct mutex doorbell_mutex;
 
void *gtt_mem;
-   uint64_t gtt_start_gpu_addr;
+   u64 gtt_start_gpu_addr;
void *gtt_start_cpu_ptr;
void *gtt_sa_bitmap;
struct mutex gtt_sa_lock;
@@ -336,9 +336,9 @@ struct kfd_dev {
bool init_complete;
 
/* Firmware versions */
-   uint16_t mec_fw_version;
-   uint16_t mec2_fw_version;
-   uint16_t sdma_fw_version;
+   u16 mec_fw_version;
+   u16 mec2_fw_version;
+   u16 sdma_fw_version;
 
/* CWSR */
bool cwsr_enabled;
@@ -346,7 +346,7 @@ struct kfd_dev {
unsigned int cwsr_isa_size;
 
/* xGMI */
-   uint64_t hive_id;
+   u64 

[PATCH] drm/amdkfd: Fix iterator used outside loop in 'kfd_add_peer_prop()'

2023-12-29 Thread Srinivasan Shanmugam
Fix the following about iterator use:
drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_topology.c:1456 kfd_add_peer_prop() 
warn: iterator used outside loop: 'iolink3'

Cc: Felix Kuehling 
Cc: Christian König 
Cc: Alex Deucher 
Signed-off-by: Srinivasan Shanmugam 
---
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 24 ---
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c 
b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
index dc7c8312e8c7..68640e46312f 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
@@ -1449,17 +1449,19 @@ static int kfd_add_peer_prop(struct kfd_topology_device 
*kdev,
/* CPU->CPU  link*/
cpu_dev = 
kfd_topology_device_by_proximity_domain(iolink1->node_to);
if (cpu_dev) {
-   list_for_each_entry(iolink3, _dev->io_link_props, 
list)
-   if (iolink3->node_to == iolink2->node_to)
-   break;
-
-   props->weight += iolink3->weight;
-   props->min_latency += iolink3->min_latency;
-   props->max_latency += iolink3->max_latency;
-   props->min_bandwidth = min(props->min_bandwidth,
-   iolink3->min_bandwidth);
-   props->max_bandwidth = min(props->max_bandwidth,
-   iolink3->max_bandwidth);
+   list_for_each_entry(iolink3, _dev->io_link_props, 
list) {
+   if (iolink3->node_to != iolink2->node_to)
+   continue;
+
+   props->weight += iolink3->weight;
+   props->min_latency += iolink3->min_latency;
+   props->max_latency += iolink3->max_latency;
+   props->min_bandwidth = min(props->min_bandwidth,
+  
iolink3->min_bandwidth);
+   props->max_bandwidth = min(props->max_bandwidth,
+  
iolink3->max_bandwidth);
+   break;
+   }
} else {
WARN(1, "CPU node not found");
}
-- 
2.34.1