Re: [PATCH v10 03/14] drm/amdgpu: add new IOCTL for usermode queue

[Sharma, Shashank]

On 02/05/2024 23:25, Alex Deucher wrote:

On Thu, May 2, 2024 at 1:27 PM Shashank Sharma  wrote:

This patch adds:
- A new IOCTL function to create and destroy
- A new structure to keep all the user queue data in one place.
- A function to generate unique index for the queue.

V1: Worked on review comments from RFC patch series:
   - Alex: Keep a list of queues, instead of single queue per process.
   - Christian: Use the queue manager instead of global ptrs,
Don't keep the queue structure in amdgpu_ctx

V2: Worked on review comments:
  - Christian:
- Formatting of text
- There is no need for queuing of userqueues, with idr in place
  - Alex:
- Remove use_doorbell, its unnecessary
- Reuse amdgpu_mqd_props for saving mqd fields

  - Code formatting and re-arrangement

V3:
  - Integration with doorbell manager

V4:
  - Accommodate MQD union related changes in UAPI (Alex)
  - Do not set the queue size twice (Bas)

V5:
  - Remove wrapper functions for queue indexing (Christian)
  - Do not save the queue id/idr in queue itself (Christian)
  - Move the idr allocation in the IP independent generic space
   (Christian)

V6:
  - Check the validity of input IP type (Christian)

V7:
  - Move uq_func from uq_mgr to adev (Alex)
  - Add missing free(queue) for error cases (Yifan)

V9:
  - Rebase

V10: Addressed review comments from Christian, and added R-B:
  - Do not initialize the local variable
  - Convert DRM_ERROR to DEBUG.

Cc: Alex Deucher 
Cc: Christian Koenig 
Reviewed-by: Christian Koenig 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c   |   1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c | 121 ++
  .../gpu/drm/amd/include/amdgpu_userqueue.h|   2 +
  3 files changed, 124 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index b52442e2d04a..551e13693100 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -2929,6 +2929,7 @@ const struct drm_ioctl_desc amdgpu_ioctls_kms[] = {
 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_VA, amdgpu_gem_va_ioctl, 
DRM_AUTH|DRM_RENDER_ALLOW),
 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_OP, amdgpu_gem_op_ioctl, 
DRM_AUTH|DRM_RENDER_ALLOW),
 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_USERPTR, amdgpu_gem_userptr_ioctl, 
DRM_AUTH|DRM_RENDER_ALLOW),
+   DRM_IOCTL_DEF_DRV(AMDGPU_USERQ, amdgpu_userq_ioctl, 
DRM_AUTH|DRM_RENDER_ALLOW),
  };

  static const struct drm_driver amdgpu_kms_driver = {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
index effc0c7c02cf..ce9b25b82e94 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
@@ -23,6 +23,127 @@
   */

  #include "amdgpu.h"
+#include "amdgpu_vm.h"
+#include "amdgpu_userqueue.h"
+
+static struct amdgpu_usermode_queue *
+amdgpu_userqueue_find(struct amdgpu_userq_mgr *uq_mgr, int qid)
+{
+   return idr_find(_mgr->userq_idr, qid);
+}
+
+static int
+amdgpu_userqueue_destroy(struct drm_file *filp, int queue_id)
+{
+   struct amdgpu_fpriv *fpriv = filp->driver_priv;
+   struct amdgpu_userq_mgr *uq_mgr = >userq_mgr;
+   struct amdgpu_device *adev = uq_mgr->adev;
+   const struct amdgpu_userq_funcs *uq_funcs;
+   struct amdgpu_usermode_queue *queue;
+
+   mutex_lock(_mgr->userq_mutex);
+
+   queue = amdgpu_userqueue_find(uq_mgr, queue_id);
+   if (!queue) {
+   DRM_DEBUG_DRIVER("Invalid queue id to destroy\n");
+   mutex_unlock(_mgr->userq_mutex);
+   return -EINVAL;
+   }
+
+   uq_funcs = adev->userq_funcs[queue->queue_type];
+   uq_funcs->mqd_destroy(uq_mgr, queue);
+   idr_remove(_mgr->userq_idr, queue_id);
+   kfree(queue);
+
+   mutex_unlock(_mgr->userq_mutex);
+   return 0;
+}
+
+static int
+amdgpu_userqueue_create(struct drm_file *filp, union drm_amdgpu_userq *args)
+{
+   struct amdgpu_fpriv *fpriv = filp->driver_priv;
+   struct amdgpu_userq_mgr *uq_mgr = >userq_mgr;
+   struct amdgpu_device *adev = uq_mgr->adev;
+   const struct amdgpu_userq_funcs *uq_funcs;
+   struct amdgpu_usermode_queue *queue;
+   int qid, r = 0;
+
+   /* Usermode queues are only supported for GFX/SDMA engines as of now */
+   if (args->in.ip_type != AMDGPU_HW_IP_GFX) {
+   DRM_ERROR("Usermode queue doesn't support IP type %u\n", 
args->in.ip_type);
+   return -EINVAL;
+   }
+
+   mutex_lock(_mgr->userq_mutex);
+
+   uq_funcs = adev->userq_funcs[args->in.ip_type];
+   if (!uq_funcs) {
+   DRM_ERROR("Usermode queue is not supported for this IP (%u)\n", 
args->in.ip_type);
+   r = -EINVAL;
+   goto unlock;
+   }
+
+   queue = kzalloc(sizeof(struct amdgpu_usermode_queue), GFP_KERNEL);
+   if (!queue) {
+   DRM_ERROR("Failed to allocate memory 

Re: [PATCH v9 14/14] drm/amdgpu: add kernel config for gfx-userqueue

[Sharma, Shashank]

On 02/05/2024 17:22, Christian König wrote:



Am 26.04.24 um 15:48 schrieb Shashank Sharma:

This patch:
- adds a kernel config option "CONFIG_DRM_AMD_USERQ_GFX"
- moves the usequeue initialization code for all IPs under
   this flag

so that the userqueue works only when the config is enabled.

Cc: Alex Deucher 
Cc: Christian Koenig 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Kconfig | 8 
  drivers/gpu/drm/amd/amdgpu/Makefile    | 8 ++--
  drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c | 4 
  drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c | 3 +++
  4 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/Kconfig 
b/drivers/gpu/drm/amd/amdgpu/Kconfig

index 22d88f8ef527..bba963527d22 100644
--- a/drivers/gpu/drm/amd/amdgpu/Kconfig
+++ b/drivers/gpu/drm/amd/amdgpu/Kconfig
@@ -80,6 +80,14 @@ config DRM_AMDGPU_WERROR
    Add -Werror to the build flags for amdgpu.ko.
    Only enable this if you are warning code for amdgpu.ko.
  +config DRM_AMDGPU_USERQ_GFX
+    bool "Enable Navi 3x gfx usermode queues"
+    depends on DRM_AMDGPU
+    default n
+    help
+  Choose this option to enable usermode queue support for GFX
+  workload submission. This feature is supported on Navi 3X 
only.


When this is for Navi 3x only I would name that 
DRM_AMDGPU_NAVI3X_USERQ instead.


And since we enable/disable GFX, Compute and SDMA I would drop "gfx" 
from the comment and description.


Apart from that the approach looks good to me.


Agree, both the review comments addressed in V10.

- Shashank


Christian.


+
  source "drivers/gpu/drm/amd/acp/Kconfig"
  source "drivers/gpu/drm/amd/display/Kconfig"
  source "drivers/gpu/drm/amd/amdkfd/Kconfig"
diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index a640bfa468ad..0b17fc1740a0 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -184,8 +184,12 @@ amdgpu-y += \
  amdgpu-y += \
  amdgpu_mes.o \
  mes_v10_1.o \
-    mes_v11_0.o \
-    mes_v11_0_userqueue.o
+    mes_v11_0.o
+
+# add GFX userqueue support
+ifneq ($(CONFIG_DRM_AMD_USERQ_GFX),)
+amdgpu-y += mes_v11_0_userqueue.o
+endif
    # add UVD block
  amdgpu-y += \
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c 
b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c

index 27b86f7fe949..8591aed9f9ab 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
@@ -1349,8 +1349,10 @@ static int gfx_v11_0_sw_init(void *handle)
  adev->gfx.mec.num_mec = 2;
  adev->gfx.mec.num_pipe_per_mec = 4;
  adev->gfx.mec.num_queue_per_pipe = 4;
+#ifdef CONFIG_DRM_AMD_USERQ_GFX
  adev->userq_funcs[AMDGPU_HW_IP_GFX] = _mes_v11_0_funcs;
  adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = 
_mes_v11_0_funcs;

+#endif
  break;
  case IP_VERSION(11, 0, 1):
  case IP_VERSION(11, 0, 4):
@@ -1362,8 +1364,10 @@ static int gfx_v11_0_sw_init(void *handle)
  adev->gfx.mec.num_mec = 1;
  adev->gfx.mec.num_pipe_per_mec = 4;
  adev->gfx.mec.num_queue_per_pipe = 4;
+#ifdef CONFIG_DRM_AMD_USERQ_GFX
  adev->userq_funcs[AMDGPU_HW_IP_GFX] = _mes_v11_0_funcs;
  adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = 
_mes_v11_0_funcs;

+#endif
  break;
  default:
  adev->gfx.me.num_me = 1;
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c 
b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c

index 90354a70c807..084059c95db6 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
@@ -1267,7 +1267,10 @@ static int sdma_v6_0_sw_init(void *handle)
  return -EINVAL;
  }
  +#ifdef CONFIG_DRM_AMD_USERQ_GFX
  adev->userq_funcs[AMDGPU_HW_IP_DMA] = _mes_v11_0_funcs;
+#endif
+
  return r;
  }

Re: [PATCH v9 14/14] drm/amdgpu: add kernel config for gfx-userqueue

[Sharma, Shashank]

On 02/05/2024 17:22, Christian König wrote:



Am 26.04.24 um 15:48 schrieb Shashank Sharma:

This patch:
- adds a kernel config option "CONFIG_DRM_AMD_USERQ_GFX"
- moves the usequeue initialization code for all IPs under
   this flag

so that the userqueue works only when the config is enabled.

Cc: Alex Deucher 
Cc: Christian Koenig 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Kconfig | 8 
  drivers/gpu/drm/amd/amdgpu/Makefile    | 8 ++--
  drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c | 4 
  drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c | 3 +++
  4 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/Kconfig 
b/drivers/gpu/drm/amd/amdgpu/Kconfig

index 22d88f8ef527..bba963527d22 100644
--- a/drivers/gpu/drm/amd/amdgpu/Kconfig
+++ b/drivers/gpu/drm/amd/amdgpu/Kconfig
@@ -80,6 +80,14 @@ config DRM_AMDGPU_WERROR
    Add -Werror to the build flags for amdgpu.ko.
    Only enable this if you are warning code for amdgpu.ko.
  +config DRM_AMDGPU_USERQ_GFX
+    bool "Enable Navi 3x gfx usermode queues"
+    depends on DRM_AMDGPU
+    default n
+    help
+  Choose this option to enable usermode queue support for GFX
+  workload submission. This feature is supported on Navi 3X 
only.


When this is for Navi 3x only I would name that 
DRM_AMDGPU_NAVI3X_USERQ instead.



Noted,
And since we enable/disable GFX, Compute and SDMA I would drop "gfx" 
from the comment and description.


Noted, I just did not want users to get confused with KFD queues, hence 
added GFX.


I will update the patch with both the changes.

- Shashank


Apart from that the approach looks good to me.

Christian.


+
  source "drivers/gpu/drm/amd/acp/Kconfig"
  source "drivers/gpu/drm/amd/display/Kconfig"
  source "drivers/gpu/drm/amd/amdkfd/Kconfig"
diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index a640bfa468ad..0b17fc1740a0 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -184,8 +184,12 @@ amdgpu-y += \
  amdgpu-y += \
  amdgpu_mes.o \
  mes_v10_1.o \
-    mes_v11_0.o \
-    mes_v11_0_userqueue.o
+    mes_v11_0.o
+
+# add GFX userqueue support
+ifneq ($(CONFIG_DRM_AMD_USERQ_GFX),)
+amdgpu-y += mes_v11_0_userqueue.o
+endif
    # add UVD block
  amdgpu-y += \
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c 
b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c

index 27b86f7fe949..8591aed9f9ab 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
@@ -1349,8 +1349,10 @@ static int gfx_v11_0_sw_init(void *handle)
  adev->gfx.mec.num_mec = 2;
  adev->gfx.mec.num_pipe_per_mec = 4;
  adev->gfx.mec.num_queue_per_pipe = 4;
+#ifdef CONFIG_DRM_AMD_USERQ_GFX
  adev->userq_funcs[AMDGPU_HW_IP_GFX] = _mes_v11_0_funcs;
  adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = 
_mes_v11_0_funcs;

+#endif
  break;
  case IP_VERSION(11, 0, 1):
  case IP_VERSION(11, 0, 4):
@@ -1362,8 +1364,10 @@ static int gfx_v11_0_sw_init(void *handle)
  adev->gfx.mec.num_mec = 1;
  adev->gfx.mec.num_pipe_per_mec = 4;
  adev->gfx.mec.num_queue_per_pipe = 4;
+#ifdef CONFIG_DRM_AMD_USERQ_GFX
  adev->userq_funcs[AMDGPU_HW_IP_GFX] = _mes_v11_0_funcs;
  adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = 
_mes_v11_0_funcs;

+#endif
  break;
  default:
  adev->gfx.me.num_me = 1;
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c 
b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c

index 90354a70c807..084059c95db6 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
@@ -1267,7 +1267,10 @@ static int sdma_v6_0_sw_init(void *handle)
  return -EINVAL;
  }
  +#ifdef CONFIG_DRM_AMD_USERQ_GFX
  adev->userq_funcs[AMDGPU_HW_IP_DMA] = _mes_v11_0_funcs;
+#endif
+
  return r;
  }

Re: [PATCH v9 11/14] drm/amdgpu: fix MES GFX mask

[Sharma, Shashank]

On 02/05/2024 17:19, Christian König wrote:

Am 26.04.24 um 15:48 schrieb Shashank Sharma:

Current MES GFX mask prevents FW to enable oversubscription. This patch
does the following:
- Fixes the mask values and adds a description for the same.
- Removes the central mask setup and makes it IP specific, as it would
   be different when the number of pipes and queues are different.

V9: introduce this patch in the series


As far as I can see this is a bug fix for existing code and should be 
pushed completely independent of the other work to amd-staging-drm-next.


Agreed, I added it here for completion of series. I had pushed this as 
single patch as well last week, I will push it accordingly.


- Shashank


Regards,
Christian.



Cc: Christian König 
Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
Signed-off-by: Arvind Yadav 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 3 ---
  drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h | 1 -
  drivers/gpu/drm/amd/amdgpu/mes_v10_1.c  | 9 +++--
  drivers/gpu/drm/amd/amdgpu/mes_v11_0.c  | 9 +++--
  4 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c

index a00cf4756ad0..b405fafc0b71 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
@@ -151,9 +151,6 @@ int amdgpu_mes_init(struct amdgpu_device *adev)
  adev->mes.compute_hqd_mask[i] = 0xc;
  }
  -    for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++)
-    adev->mes.gfx_hqd_mask[i] = i ? 0 : 0xfffe;
-
  for (i = 0; i < AMDGPU_MES_MAX_SDMA_PIPES; i++) {
  if (amdgpu_ip_version(adev, SDMA0_HWIP, 0) <
  IP_VERSION(6, 0, 0))
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h

index 4c8fc3117ef8..598556619337 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h
@@ -110,7 +110,6 @@ struct amdgpu_mes {
  uint32_t    vmid_mask_gfxhub;
  uint32_t    vmid_mask_mmhub;
  uint32_t compute_hqd_mask[AMDGPU_MES_MAX_COMPUTE_PIPES];
-    uint32_t gfx_hqd_mask[AMDGPU_MES_MAX_GFX_PIPES];
  uint32_t sdma_hqd_mask[AMDGPU_MES_MAX_SDMA_PIPES];
  uint32_t aggregated_doorbells[AMDGPU_MES_PRIORITY_NUM_LEVELS];
  uint32_t    sch_ctx_offs;
diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v10_1.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v10_1.c

index 1e5ad1e08d2a..4d1121d1a1e7 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v10_1.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v10_1.c
@@ -290,8 +290,13 @@ static int mes_v10_1_set_hw_resources(struct 
amdgpu_mes *mes)

  mes_set_hw_res_pkt.compute_hqd_mask[i] =
  mes->compute_hqd_mask[i];
  -    for (i = 0; i < MAX_GFX_PIPES; i++)
-    mes_set_hw_res_pkt.gfx_hqd_mask[i] = mes->gfx_hqd_mask[i];
+    /*
+ * GFX pipe 0 queue 0 is being used by kernel
+ * Set GFX pipe 0 queue 1 for MES scheduling
+ * GFX pipe 1 can't be used for MES due to HW limitation.
+ */
+    mes_set_hw_res_pkt.gfx_hqd_mask[0] = 0x2;
+    mes_set_hw_res_pkt.gfx_hqd_mask[1] = 0;
    for (i = 0; i < MAX_SDMA_PIPES; i++)
  mes_set_hw_res_pkt.sdma_hqd_mask[i] = mes->sdma_hqd_mask[i];
diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c

index 63f281a9984d..feb7fa2c304c 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
@@ -387,8 +387,13 @@ static int mes_v11_0_set_hw_resources(struct 
amdgpu_mes *mes)

  mes_set_hw_res_pkt.compute_hqd_mask[i] =
  mes->compute_hqd_mask[i];
  -    for (i = 0; i < MAX_GFX_PIPES; i++)
-    mes_set_hw_res_pkt.gfx_hqd_mask[i] = mes->gfx_hqd_mask[i];
+    /*
+ * GFX pipe 0 queue 0 is being used by kernel
+ * Set GFX pipe 0 queue 1 for MES scheduling
+ * GFX pipe 1 can't be used for MES due to HW limitation.
+ */
+    mes_set_hw_res_pkt.gfx_hqd_mask[0] = 0x2;
+    mes_set_hw_res_pkt.gfx_hqd_mask[1] = 0;
    for (i = 0; i < MAX_SDMA_PIPES; i++)
  mes_set_hw_res_pkt.sdma_hqd_mask[i] = mes->sdma_hqd_mask[i];

Re: [PATCH v9 08/14] drm/amdgpu: map wptr BO into GART

[Sharma, Shashank]

On 02/05/2024 17:18, Christian König wrote:

Am 26.04.24 um 15:48 schrieb Shashank Sharma:

To support oversubscription, MES FW expects WPTR BOs to
be mapped into GART, before they are submitted to usermode
queues. This patch adds a function for the same.

V4: fix the wptr value before mapping lookup (Bas, Christian).

V5: Addressed review comments from Christian:
 - Either pin object or allocate from GART, but not both.
 - All the handling must be done with the VM locks held.

V7: Addressed review comments from Christian:
 - Do not take vm->eviction_lock
 - Use amdgpu_bo_gpu_offset to get the wptr_bo GPU offset

V8: Rebase
V9: Changed the function names from gfx_v11* to mes_v11*

Cc: Alex Deucher 
Cc: Christian Koenig 
Signed-off-by: Shashank Sharma 
Signed-off-by: Arvind Yadav 


The patch itself looks good, but this really need the eviction fence 
to work properly.


Otherwise it can be that the BO mapped into the GART is evicted at 
some point.



Noted, eviction fences will be following up soon.

- Shashank



Christian.


---
  .../gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c  | 77 +++
  .../gpu/drm/amd/include/amdgpu_userqueue.h    |  1 +
  2 files changed, 78 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c

index 8d2cd61af26b..37b80626e792 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
@@ -30,6 +30,74 @@
  #define AMDGPU_USERQ_PROC_CTX_SZ PAGE_SIZE
  #define AMDGPU_USERQ_GANG_CTX_SZ PAGE_SIZE
  +static int
+mes_v11_0_map_gtt_bo_to_gart(struct amdgpu_device *adev, struct 
amdgpu_bo *bo)

+{
+    int ret;
+
+    ret = amdgpu_bo_reserve(bo, true);
+    if (ret) {
+    DRM_ERROR("Failed to reserve bo. ret %d\n", ret);
+    goto err_reserve_bo_failed;
+    }
+
+    ret = amdgpu_ttm_alloc_gart(>tbo);
+    if (ret) {
+    DRM_ERROR("Failed to bind bo to GART. ret %d\n", ret);
+    goto err_map_bo_gart_failed;
+    }
+
+    amdgpu_bo_unreserve(bo);
+    bo = amdgpu_bo_ref(bo);
+
+    return 0;
+
+err_map_bo_gart_failed:
+    amdgpu_bo_unreserve(bo);
+err_reserve_bo_failed:
+    return ret;
+}
+
+static int
+mes_v11_0_create_wptr_mapping(struct amdgpu_userq_mgr *uq_mgr,
+  struct amdgpu_usermode_queue *queue,
+  uint64_t wptr)
+{
+    struct amdgpu_device *adev = uq_mgr->adev;
+    struct amdgpu_bo_va_mapping *wptr_mapping;
+    struct amdgpu_vm *wptr_vm;
+    struct amdgpu_userq_obj *wptr_obj = >wptr_obj;
+    int ret;
+
+    wptr_vm = queue->vm;
+    ret = amdgpu_bo_reserve(wptr_vm->root.bo, false);
+    if (ret)
+    return ret;
+
+    wptr &= AMDGPU_GMC_HOLE_MASK;
+    wptr_mapping = amdgpu_vm_bo_lookup_mapping(wptr_vm, wptr >> 
PAGE_SHIFT);

+    amdgpu_bo_unreserve(wptr_vm->root.bo);
+    if (!wptr_mapping) {
+    DRM_ERROR("Failed to lookup wptr bo\n");
+    return -EINVAL;
+    }
+
+    wptr_obj->obj = wptr_mapping->bo_va->base.bo;
+    if (wptr_obj->obj->tbo.base.size > PAGE_SIZE) {
+    DRM_ERROR("Requested GART mapping for wptr bo larger than 
one page\n");

+    return -EINVAL;
+    }
+
+    ret = mes_v11_0_map_gtt_bo_to_gart(adev, wptr_obj->obj);
+    if (ret) {
+    DRM_ERROR("Failed to map wptr bo to GART\n");
+    return ret;
+    }
+
+    queue->wptr_obj.gpu_addr = 
amdgpu_bo_gpu_offset_no_check(wptr_obj->obj);

+    return 0;
+}
+
  static int mes_v11_0_userq_map(struct amdgpu_userq_mgr *uq_mgr,
 struct amdgpu_usermode_queue *queue,
 struct amdgpu_mqd_prop *userq_props)
@@ -61,6 +129,7 @@ static int mes_v11_0_userq_map(struct 
amdgpu_userq_mgr *uq_mgr,

  queue_input.queue_size = userq_props->queue_size >> 2;
  queue_input.doorbell_offset = userq_props->doorbell_index;
  queue_input.page_table_base_addr = 
amdgpu_gmc_pd_addr(queue->vm->root.bo);

+    queue_input.wptr_mc_addr = queue->wptr_obj.gpu_addr;
    amdgpu_mes_lock(>mes);
  r = adev->mes.funcs->add_hw_queue(>mes, _input);
@@ -187,6 +256,13 @@ static int mes_v11_0_userq_mqd_create(struct 
amdgpu_userq_mgr *uq_mgr,

  goto free_mqd;
  }
  +    /* FW expects WPTR BOs to be mapped into GART */
+    r = mes_v11_0_create_wptr_mapping(uq_mgr, queue, 
userq_props->wptr_gpu_addr);

+    if (r) {
+    DRM_ERROR("Failed to create WPTR mapping\n");
+    goto free_ctx;
+    }
+
  /* Map userqueue into FW using MES */
  r = mes_v11_0_userq_map(uq_mgr, queue, userq_props);
  if (r) {
@@ -216,6 +292,7 @@ mes_v11_0_userq_mqd_destroy(struct 
amdgpu_userq_mgr *uq_mgr,

  struct amdgpu_usermode_queue *queue)
  {
  mes_v11_0_userq_unmap(uq_mgr, queue);
+    amdgpu_bo_unref(>wptr_obj.obj);
  amdgpu_userqueue_destroy_object(uq_mgr, >fw_obj);
  kfree(queue->userq_prop);
  amdgpu_userqueue_destroy_object(uq_mgr, >mqd);
diff --git a/drivers/gpu/drm/amd/include/amdgpu_userqueue.h 

Re: [PATCH v9 05/14] drm/amdgpu: create MES-V11 usermode queue for GFX

[Sharma, Shashank]

On 02/05/2024 17:14, Christian König wrote:



Am 26.04.24 um 15:48 schrieb Shashank Sharma:

A Memory queue descriptor (MQD) of a userqueue defines it in
the hw's context. As MQD format can vary between different
graphics IPs, we need gfx GEN specific handlers to create MQDs.

This patch:
- Adds a new file which will be used for MES based userqueue
   functions targeting GFX and SDMA IP.
- Introduces MQD handler functions for the usermode queues.
- Adds new functions to create and destroy userqueue MQD for
   MES-V11 for GFX IP.

V1: Worked on review comments from Alex:
 - Make MQD functions GEN and IP specific

V2: Worked on review comments from Alex:
 - Reuse the existing adev->mqd[ip] for MQD creation
 - Formatting and arrangement of code

V3:
 - Integration with doorbell manager

V4: Review comments addressed:
 - Do not create a new file for userq, reuse gfx_v11_0.c (Alex)
 - Align name of structure members (Luben)
 - Don't break up the Cc tag list and the Sob tag list in commit
   message (Luben)
V5:
    - No need to reserve the bo for MQD (Christian).
    - Some more changes to support IP specific MQD creation.

V6:
    - Add a comment reminding us to replace the 
amdgpu_bo_create_kernel()
  calls while creating MQD object to amdgpu_bo_create() once 
eviction

  fences are ready (Christian).

V7:
    - Re-arrange userqueue functions in adev instead of uq_mgr (Alex)
    - Use memdup_user instead of copy_from_user (Christian)

V9:
    - Moved userqueue code from gfx_v11_0.c to new file mes_v11_0.c so
  that it can be reused for SDMA userqueues as well (Shashank, Alex)

Cc: Alex Deucher 
Cc: Christian Koenig 
Signed-off-by: Shashank Sharma 
Signed-off-by: Arvind Yadav 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |   3 +-
  drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c    |   4 +
  .../gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c  | 110 ++
  3 files changed, 116 insertions(+), 1 deletion(-)
  create mode 100644 drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index 05a2d1714070..a640bfa468ad 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -184,7 +184,8 @@ amdgpu-y += \
  amdgpu-y += \
  amdgpu_mes.o \
  mes_v10_1.o \
-    mes_v11_0.o
+    mes_v11_0.o \
+    mes_v11_0_userqueue.o


Do we really need a new C file for this or could we put the two 
functions into mes_v11_0.c as well?


Apart from that it looks correct to me, but I'm really not that deep 
inside the code at the moment.


Actually, this patch adds these two functions, and then the upcoming 
patches add other multiple functions to create/destroy FW objects, 
map/unmap_queue, handle doorbell and map wptr BO on top of these. So 
when we look at it in the end, its probably fine :).


- Shashank


Regards,
Christian.


    # add UVD block
  amdgpu-y += \
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c 
b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c

index f7325b02a191..525bd0f4d3f7 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
@@ -1331,6 +1331,8 @@ static int 
gfx_v11_0_rlc_backdoor_autoload_enable(struct amdgpu_device *adev)

  return 0;
  }
  +extern const struct amdgpu_userq_funcs userq_mes_v11_0_funcs;
+
  static int gfx_v11_0_sw_init(void *handle)
  {
  int i, j, k, r, ring_id = 0;
@@ -1347,6 +1349,7 @@ static int gfx_v11_0_sw_init(void *handle)
  adev->gfx.mec.num_mec = 2;
  adev->gfx.mec.num_pipe_per_mec = 4;
  adev->gfx.mec.num_queue_per_pipe = 4;
+    adev->userq_funcs[AMDGPU_HW_IP_GFX] = _mes_v11_0_funcs;
  break;
  case IP_VERSION(11, 0, 1):
  case IP_VERSION(11, 0, 4):
@@ -1358,6 +1361,7 @@ static int gfx_v11_0_sw_init(void *handle)
  adev->gfx.mec.num_mec = 1;
  adev->gfx.mec.num_pipe_per_mec = 4;
  adev->gfx.mec.num_queue_per_pipe = 4;
+    adev->userq_funcs[AMDGPU_HW_IP_GFX] = _mes_v11_0_funcs;
  break;
  default:
  adev->gfx.me.num_me = 1;
diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c

new file mode 100644
index ..9e7dee77d344
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
@@ -0,0 +1,110 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2024 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person 
obtaining a
+ * copy of this software and associated documentation files (the 
"Software"),
+ * to deal in the Software without restriction, including without 
limitation
+ * the rights to use, copy, modify, merge, publish, distribute, 
sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom 
the

+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this 

Re: [PATCH v9 13/14] drm/amdgpu: enable compute/gfx usermode queue

[Sharma, Shashank]

On 02/05/2024 16:10, Alex Deucher wrote:

On Thu, May 2, 2024 at 1:51 AM Sharma, Shashank  wrote:


On 01/05/2024 22:44, Alex Deucher wrote:

On Fri, Apr 26, 2024 at 10:27 AM Shashank Sharma
 wrote:

From: Arvind Yadav 

This patch does the necessary changes required to
enable compute workload support using the existing
usermode queues infrastructure.

Cc: Alex Deucher 
Cc: Christian Koenig 
Signed-off-by: Arvind Yadav 
Signed-off-by: Shashank Sharma 
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c|  3 ++-
   drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c   |  2 ++
   drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c | 10 +-
   include/uapi/drm/amdgpu_drm.h|  1 +
   4 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
index e516487e8db9..78d34fa7a0b9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
@@ -189,7 +189,8 @@ amdgpu_userqueue_create(struct drm_file *filp, union 
drm_amdgpu_userq *args)
  int qid, r = 0;

  /* Usermode queues are only supported for GFX/SDMA engines as of now 
*/
-   if (args->in.ip_type != AMDGPU_HW_IP_GFX && args->in.ip_type != 
AMDGPU_HW_IP_DMA) {
+   if (args->in.ip_type != AMDGPU_HW_IP_GFX && args->in.ip_type != 
AMDGPU_HW_IP_DMA
+   && args->in.ip_type != AMDGPU_HW_IP_COMPUTE) {
  DRM_ERROR("Usermode queue doesn't support IP type %u\n", 
args->in.ip_type);
  return -EINVAL;
  }
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c 
b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
index 525bd0f4d3f7..27b86f7fe949 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
@@ -1350,6 +1350,7 @@ static int gfx_v11_0_sw_init(void *handle)
  adev->gfx.mec.num_pipe_per_mec = 4;
  adev->gfx.mec.num_queue_per_pipe = 4;
  adev->userq_funcs[AMDGPU_HW_IP_GFX] = _mes_v11_0_funcs;
+   adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = 
_mes_v11_0_funcs;
  break;
  case IP_VERSION(11, 0, 1):
  case IP_VERSION(11, 0, 4):
@@ -1362,6 +1363,7 @@ static int gfx_v11_0_sw_init(void *handle)
  adev->gfx.mec.num_pipe_per_mec = 4;
  adev->gfx.mec.num_queue_per_pipe = 4;
  adev->userq_funcs[AMDGPU_HW_IP_GFX] = _mes_v11_0_funcs;
+   adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = 
_mes_v11_0_funcs;
  break;
  default:
  adev->gfx.me.num_me = 1;
diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
index a5e270eda37b..d61d80f86003 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
@@ -183,7 +183,8 @@ static int mes_v11_0_userq_create_ctx_space(struct 
amdgpu_userq_mgr *uq_mgr,
  }

  /* We don't need to set other FW objects for SDMA queues */
-   if (queue->queue_type == AMDGPU_HW_IP_DMA)
+   if ((queue->queue_type == AMDGPU_HW_IP_DMA) ||
+   (queue->queue_type == AMDGPU_HW_IP_COMPUTE))
  return 0;

  /* Shadow and GDS objects come directly from userspace */
@@ -246,6 +247,13 @@ static int mes_v11_0_userq_mqd_create(struct 
amdgpu_userq_mgr *uq_mgr,
  userq_props->use_doorbell = true;
  userq_props->doorbell_index = queue->doorbell_index;

+   if (queue->queue_type == AMDGPU_HW_IP_COMPUTE) {
+   userq_props->eop_gpu_addr = mqd_user->eop_va;
+   userq_props->hqd_pipe_priority = AMDGPU_GFX_PIPE_PRIO_NORMAL;
+   userq_props->hqd_queue_priority = 
AMDGPU_GFX_QUEUE_PRIORITY_MINIMUM;
+   userq_props->hqd_active = false;
+   }
+
  queue->userq_prop = userq_props;

  r = mqd_hw_default->init_mqd(adev, (void *)queue->mqd.cpu_ptr, 
userq_props);
diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
index 22f56a30f7cb..676792ad3618 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -375,6 +375,7 @@ struct drm_amdgpu_userq_mqd {
   * sized.
   */
  __u64   csa_va;
+   __u64   eop_va;
   };

Let's add a new mqd descriptor for compute since it's different from
gfx and sdma.

the only different thing is this object (vs csa and gds objects), apart
from that, the mqd is the same as they all are MES based. Am I missing
something here ?

The scheduling entity is irrelevant.  The mqd is defined by the engine
itself.  E.g., v11_structs.h.  Gfx has one set of requirements,
compute has different ones, and SDMA has different ones.  VPE and VCN
also have mqds.  When 

Re: [PATCH v9 12/14] drm/amdgpu: enable SDMA usermode queues

[Sharma, Shashank]

On 02/05/2024 15:55, Alex Deucher wrote:

On Thu, May 2, 2024 at 1:47 AM Sharma, Shashank  wrote:


On 01/05/2024 22:41, Alex Deucher wrote:

On Fri, Apr 26, 2024 at 10:27 AM Shashank Sharma
 wrote:

This patch does necessary modifications to enable the SDMA
usermode queues using the existing userqueue infrastructure.

V9: introduced this patch in the series

Cc: Christian König 
Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
Signed-off-by: Arvind Yadav 
Signed-off-by: Srinivasan Shanmugam 
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c| 2 +-
   drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c | 4 
   drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c   | 3 +++
   3 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
index 781283753804..e516487e8db9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
@@ -189,7 +189,7 @@ amdgpu_userqueue_create(struct drm_file *filp, union 
drm_amdgpu_userq *args)
  int qid, r = 0;

  /* Usermode queues are only supported for GFX/SDMA engines as of now 
*/
-   if (args->in.ip_type != AMDGPU_HW_IP_GFX) {
+   if (args->in.ip_type != AMDGPU_HW_IP_GFX && args->in.ip_type != 
AMDGPU_HW_IP_DMA) {
  DRM_ERROR("Usermode queue doesn't support IP type %u\n", 
args->in.ip_type);
  return -EINVAL;
  }
diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
index a6c3037d2d1f..a5e270eda37b 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
@@ -182,6 +182,10 @@ static int mes_v11_0_userq_create_ctx_space(struct 
amdgpu_userq_mgr *uq_mgr,
  return r;
  }

+   /* We don't need to set other FW objects for SDMA queues */
+   if (queue->queue_type == AMDGPU_HW_IP_DMA)
+   return 0;
+
  /* Shadow and GDS objects come directly from userspace */
  mqd->shadow_base_lo = mqd_user->shadow_va & 0xFFFC;
  mqd->shadow_base_hi = upper_32_bits(mqd_user->shadow_va);
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c 
b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
index 361835a61f2e..90354a70c807 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
@@ -1225,6 +1225,8 @@ static int sdma_v6_0_early_init(void *handle)
  return 0;
   }

+extern const struct amdgpu_userq_funcs userq_mes_v11_0_funcs;

Can you include the header rather than adding an extern?

Noted,

+
   static int sdma_v6_0_sw_init(void *handle)
   {
  struct amdgpu_ring *ring;
@@ -1265,6 +1267,7 @@ static int sdma_v6_0_sw_init(void *handle)
  return -EINVAL;
  }

+   adev->userq_funcs[AMDGPU_HW_IP_DMA] = _mes_v11_0_funcs;
  return r;
   }

I think we need a new mqd descriptor in amdgpu_drm.h as well since the
sdma metadata is different from gfx and compute.

Can you please elaborate on this ? AFAIK SDMA queue doesn't need any
specific metadata objects (like GFX).

Right.  I want to make it clear in the IOCTL interface what buffers
are required for which ring types.  E.g., UMD might allocate a shadow
buffer for SDMA, but they don't need it so there is no need to
allocate it.  If we have separate mqd structures for every ring type,
it makes it clear which additional buffers are needed for which ring
types.


Agree, it makes sense.

- Shashank


Alex


- Shashank


Alex


--
2.43.2

Re: [PATCH v9 08/14] drm/amdgpu: map wptr BO into GART

[Sharma, Shashank]

On 02/05/2024 15:06, Kasiviswanathan, Harish wrote:

[AMD Official Use Only - General]

-Original Message-
From: amd-gfx  On Behalf Of Sharma, 
Shashank
Sent: Thursday, May 2, 2024 1:32 AM
To: Alex Deucher 
Cc: amd-gfx@lists.freedesktop.org; Yadav, Arvind ; Deucher, Alexander 
; Koenig, Christian 
Subject: Re: [PATCH v9 08/14] drm/amdgpu: map wptr BO into GART


On 01/05/2024 23:36, Alex Deucher wrote:

On Fri, Apr 26, 2024 at 9:57 AM Shashank Sharma  wrote:

To support oversubscription, MES FW expects WPTR BOs to
be mapped into GART, before they are submitted to usermode
queues. This patch adds a function for the same.

V4: fix the wptr value before mapping lookup (Bas, Christian).

V5: Addressed review comments from Christian:
  - Either pin object or allocate from GART, but not both.
  - All the handling must be done with the VM locks held.

V7: Addressed review comments from Christian:
  - Do not take vm->eviction_lock
  - Use amdgpu_bo_gpu_offset to get the wptr_bo GPU offset

V8: Rebase
V9: Changed the function names from gfx_v11* to mes_v11*

Cc: Alex Deucher 
Cc: Christian Koenig 
Signed-off-by: Shashank Sharma 
Signed-off-by: Arvind Yadav 
---
   .../gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c  | 77 +++
   .../gpu/drm/amd/include/amdgpu_userqueue.h|  1 +
   2 files changed, 78 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
index 8d2cd61af26b..37b80626e792 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
@@ -30,6 +30,74 @@
   #define AMDGPU_USERQ_PROC_CTX_SZ PAGE_SIZE
   #define AMDGPU_USERQ_GANG_CTX_SZ PAGE_SIZE

+static int
+mes_v11_0_map_gtt_bo_to_gart(struct amdgpu_device *adev, struct amdgpu_bo *bo)
+{
+   int ret;
+
+   ret = amdgpu_bo_reserve(bo, true);
+   if (ret) {
+   DRM_ERROR("Failed to reserve bo. ret %d\n", ret);
+   goto err_reserve_bo_failed;
+   }
+
+   ret = amdgpu_ttm_alloc_gart(>tbo);
+   if (ret) {
+   DRM_ERROR("Failed to bind bo to GART. ret %d\n", ret);
+   goto err_map_bo_gart_failed;
+   }
+
+   amdgpu_bo_unreserve(bo);
+   bo = amdgpu_bo_ref(bo);
+
+   return 0;
+
+err_map_bo_gart_failed:
+   amdgpu_bo_unreserve(bo);
+err_reserve_bo_failed:
+   return ret;
+}
+

There is a very similar function amdgpu_amdkfd_map_gtt_bo_to_gart(). Is it 
possible to unify. Also, adev parameter in the above function is confusing. 
This was also removed from amdgpu_amdkfd_map_gtt_bo_to_gart(). It looks like bo 
is mapped to gart of adev, however it doesn't have to be. It is mapped to the 
gart to which bo is associated.


I don't think unification makes much sense here, but I agree that adev 
can be removed from the input args. I will update this.


- Shashank


+static int
+mes_v11_0_create_wptr_mapping(struct amdgpu_userq_mgr *uq_mgr,
+ struct amdgpu_usermode_queue *queue,
+ uint64_t wptr)
+{
+   struct amdgpu_device *adev = uq_mgr->adev;
+   struct amdgpu_bo_va_mapping *wptr_mapping;
+   struct amdgpu_vm *wptr_vm;
+   struct amdgpu_userq_obj *wptr_obj = >wptr_obj;
+   int ret;
+
+   wptr_vm = queue->vm;
+   ret = amdgpu_bo_reserve(wptr_vm->root.bo, false);
+   if (ret)
+   return ret;
+
+   wptr &= AMDGPU_GMC_HOLE_MASK;
+   wptr_mapping = amdgpu_vm_bo_lookup_mapping(wptr_vm, wptr >> PAGE_SHIFT);
+   amdgpu_bo_unreserve(wptr_vm->root.bo);
+   if (!wptr_mapping) {
+   DRM_ERROR("Failed to lookup wptr bo\n");
+   return -EINVAL;
+   }
+
+   wptr_obj->obj = wptr_mapping->bo_va->base.bo;
+   if (wptr_obj->obj->tbo.base.size > PAGE_SIZE) {
+   DRM_ERROR("Requested GART mapping for wptr bo larger than one 
page\n");
+   return -EINVAL;
+   }
+
+   ret = mes_v11_0_map_gtt_bo_to_gart(adev, wptr_obj->obj);
+   if (ret) {
+   DRM_ERROR("Failed to map wptr bo to GART\n");
+   return ret;
+   }
+
+   queue->wptr_obj.gpu_addr = amdgpu_bo_gpu_offset_no_check(wptr_obj->obj);

The wptr virtual address from the user may not be at offset 0 from the
start of the object.  We should add the offset to the base vmid0 GPU
address.

can you please elaborate a bit here ? wptr_obj->obj is already mapped to
gart, do we still need this ?

- Shashank


Alex


+   return 0;
+}
+
   static int mes_v11_0_userq_map(struct amdgpu_userq_mgr *uq_mgr,
 struct amdgpu_usermode_queue *queue,
 struct amdgpu_mqd_prop *userq_props)
@@ -61,6 +129,7 @@ static int mes_v11_0_userq_map(struct amdgpu_userq_mgr 
*uq_mgr,
  queue_input

Re: [PATCH v9 01/14] drm/amdgpu: UAPI for user queue management

[Sharma, Shashank]

On 02/05/2024 07:23, Sharma, Shashank wrote:

Hey Alex,

On 01/05/2024 22:39, Alex Deucher wrote:

On Fri, Apr 26, 2024 at 10:07 AM Shashank Sharma
 wrote:

From: Alex Deucher 

This patch intorduces new UAPI/IOCTL for usermode graphics
queue. The userspace app will fill this structure and request
the graphics driver to add a graphics work queue for it. The
output of this UAPI is a queue id.

This UAPI maps the queue into GPU, so the graphics app can start
submitting work to the queue as soon as the call returns.

V2: Addressed review comments from Alex and Christian
 - Make the doorbell offset's comment clearer
 - Change the output parameter name to queue_id

V3: Integration with doorbell manager

V4:
 - Updated the UAPI doc (Pierre-Eric)
 - Created a Union for engine specific MQDs (Alex)
 - Added Christian's R-B
V5:
 - Add variables for GDS and CSA in MQD structure (Alex)
 - Make MQD data a ptr-size pair instead of union (Alex)

V9:
    - renamed struct drm_amdgpu_userq_mqd_gfx_v11 to struct
  drm_amdgpu_userq_mqd as its being used for SDMA and
  compute queues as well

Cc: Alex Deucher 
Cc: Christian Koenig 
Reviewed-by: Christian König 
Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  include/uapi/drm/amdgpu_drm.h | 110 
++

  1 file changed, 110 insertions(+)

diff --git a/include/uapi/drm/amdgpu_drm.h 
b/include/uapi/drm/amdgpu_drm.h

index 96e32dafd4f0..22f56a30f7cb 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -54,6 +54,7 @@ extern "C" {
  #define DRM_AMDGPU_VM  0x13
  #define DRM_AMDGPU_FENCE_TO_HANDLE 0x14
  #define DRM_AMDGPU_SCHED   0x15
+#define DRM_AMDGPU_USERQ   0x16

  #define DRM_IOCTL_AMDGPU_GEM_CREATE DRM_IOWR(DRM_COMMAND_BASE + 
DRM_AMDGPU_GEM_CREATE, union drm_amdgpu_gem_create)
  #define DRM_IOCTL_AMDGPU_GEM_MMAP DRM_IOWR(DRM_COMMAND_BASE + 
DRM_AMDGPU_GEM_MMAP, union drm_amdgpu_gem_mmap)

@@ -71,6 +72,7 @@ extern "C" {
  #define DRM_IOCTL_AMDGPU_VM DRM_IOWR(DRM_COMMAND_BASE + 
DRM_AMDGPU_VM, union drm_amdgpu_vm)
  #define DRM_IOCTL_AMDGPU_FENCE_TO_HANDLE DRM_IOWR(DRM_COMMAND_BASE 
+ DRM_AMDGPU_FENCE_TO_HANDLE, union drm_amdgpu_fence_to_handle)
  #define DRM_IOCTL_AMDGPU_SCHED DRM_IOW(DRM_COMMAND_BASE + 
DRM_AMDGPU_SCHED, union drm_amdgpu_sched)
+#define DRM_IOCTL_AMDGPU_USERQ DRM_IOW(DRM_COMMAND_BASE + 
DRM_AMDGPU_USERQ, union drm_amdgpu_userq)


  /**
   * DOC: memory domains
@@ -317,6 +319,114 @@ union drm_amdgpu_ctx {
 union drm_amdgpu_ctx_out out;
  };

+/* user queue IOCTL */
+#define AMDGPU_USERQ_OP_CREATE 1
+#define AMDGPU_USERQ_OP_FREE   2
+
+/* Flag to indicate secure buffer related workload, unused for now */
+#define AMDGPU_USERQ_MQD_FLAGS_SECURE  (1 << 0)
+/* Flag to indicate AQL workload, unused for now */
+#define AMDGPU_USERQ_MQD_FLAGS_AQL (1 << 1)
+
+/*
+ * MQD (memory queue descriptor) is a set of parameters which allow
+ * the GPU to uniquely define and identify a usermode queue. This
+ * structure defines the MQD for GFX-V11 IP ver 0.
+ */
+struct drm_amdgpu_userq_mqd {

Maybe rename this to drm_amdgpu_gfx_userq_mqd since it's gfx specific.
Then we can add different MQDs for SDMA, compute, etc. as they have
different metadata.  E.g., the shadow and CSA are gfx only.



Actually this was named drm_amdgpu_userq_mqd_gfx_v11_0 until the last 
patchset, but then I realized that apart from the objects (gds/shadow 
va) nothing is gfx specific, its actually required for every userqueue 
IP which is MES based, so I thought it would be an overkill to create 
multiple structures for almost the same data. If you feel strong about 
this, I can change it again.


- Shashank



Please ignore my last comment, I understand what you are mentioning, and 
I have reformatted the patches accordingly. Now, I am keeping everything 
reqd for MES in one basic struture (drm_amdgpu_userq_in) and creating  
drm_amdgpu_userq_mqd_gfx_v11 for GFX specific things (like CSA, Shadow 
and GDS areas). Now there will be one separate patch which will enabled 
GFX_IP on MES code, just like how we have separate patches for SDMA and 
Compute IP in this series.  I will send the V10 patches with this 
reformatting in some time.


- Shashank




Alex



+   /**
+    * @queue_va: Virtual address of the GPU memory which holds 
the queue

+    * object. The queue holds the workload packets.
+    */
+   __u64   queue_va;
+   /**
+    * @queue_size: Size of the queue in bytes, this needs to be 
256-byte

+    * aligned.
+    */
+   __u64   queue_size;
+   /**
+    * @rptr_va : Virtual address of the GPU memory which holds 
the ring RPTR.
+    * This object must be at least 8 byte in size and aligned 
to 8-byte offset.

+    */
+   __u64   rptr_va;
+   /**
+    * @wptr_va : Virtual address of the GPU memory which holds 
the ring W

Re: [PATCH v9 13/14] drm/amdgpu: enable compute/gfx usermode queue

[Sharma, Shashank]

On 01/05/2024 22:44, Alex Deucher wrote:

On Fri, Apr 26, 2024 at 10:27 AM Shashank Sharma
 wrote:

From: Arvind Yadav 

This patch does the necessary changes required to
enable compute workload support using the existing
usermode queues infrastructure.

Cc: Alex Deucher 
Cc: Christian Koenig 
Signed-off-by: Arvind Yadav 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c|  3 ++-
  drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c   |  2 ++
  drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c | 10 +-
  include/uapi/drm/amdgpu_drm.h|  1 +
  4 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
index e516487e8db9..78d34fa7a0b9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
@@ -189,7 +189,8 @@ amdgpu_userqueue_create(struct drm_file *filp, union 
drm_amdgpu_userq *args)
 int qid, r = 0;

 /* Usermode queues are only supported for GFX/SDMA engines as of now */
-   if (args->in.ip_type != AMDGPU_HW_IP_GFX && args->in.ip_type != 
AMDGPU_HW_IP_DMA) {
+   if (args->in.ip_type != AMDGPU_HW_IP_GFX && args->in.ip_type != 
AMDGPU_HW_IP_DMA
+   && args->in.ip_type != AMDGPU_HW_IP_COMPUTE) {
 DRM_ERROR("Usermode queue doesn't support IP type %u\n", 
args->in.ip_type);
 return -EINVAL;
 }
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c 
b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
index 525bd0f4d3f7..27b86f7fe949 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
@@ -1350,6 +1350,7 @@ static int gfx_v11_0_sw_init(void *handle)
 adev->gfx.mec.num_pipe_per_mec = 4;
 adev->gfx.mec.num_queue_per_pipe = 4;
 adev->userq_funcs[AMDGPU_HW_IP_GFX] = _mes_v11_0_funcs;
+   adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = 
_mes_v11_0_funcs;
 break;
 case IP_VERSION(11, 0, 1):
 case IP_VERSION(11, 0, 4):
@@ -1362,6 +1363,7 @@ static int gfx_v11_0_sw_init(void *handle)
 adev->gfx.mec.num_pipe_per_mec = 4;
 adev->gfx.mec.num_queue_per_pipe = 4;
 adev->userq_funcs[AMDGPU_HW_IP_GFX] = _mes_v11_0_funcs;
+   adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = 
_mes_v11_0_funcs;
 break;
 default:
 adev->gfx.me.num_me = 1;
diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
index a5e270eda37b..d61d80f86003 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
@@ -183,7 +183,8 @@ static int mes_v11_0_userq_create_ctx_space(struct 
amdgpu_userq_mgr *uq_mgr,
 }

 /* We don't need to set other FW objects for SDMA queues */
-   if (queue->queue_type == AMDGPU_HW_IP_DMA)
+   if ((queue->queue_type == AMDGPU_HW_IP_DMA) ||
+   (queue->queue_type == AMDGPU_HW_IP_COMPUTE))
 return 0;

 /* Shadow and GDS objects come directly from userspace */
@@ -246,6 +247,13 @@ static int mes_v11_0_userq_mqd_create(struct 
amdgpu_userq_mgr *uq_mgr,
 userq_props->use_doorbell = true;
 userq_props->doorbell_index = queue->doorbell_index;

+   if (queue->queue_type == AMDGPU_HW_IP_COMPUTE) {
+   userq_props->eop_gpu_addr = mqd_user->eop_va;
+   userq_props->hqd_pipe_priority = AMDGPU_GFX_PIPE_PRIO_NORMAL;
+   userq_props->hqd_queue_priority = 
AMDGPU_GFX_QUEUE_PRIORITY_MINIMUM;
+   userq_props->hqd_active = false;
+   }
+
 queue->userq_prop = userq_props;

 r = mqd_hw_default->init_mqd(adev, (void *)queue->mqd.cpu_ptr, 
userq_props);
diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
index 22f56a30f7cb..676792ad3618 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -375,6 +375,7 @@ struct drm_amdgpu_userq_mqd {
  * sized.
  */
 __u64   csa_va;
+   __u64   eop_va;
  };

Let's add a new mqd descriptor for compute since it's different from
gfx and sdma.
the only different thing is this object (vs csa and gds objects), apart 
from that, the mqd is the same as they all are MES based. Am I missing 
something here ?


Also, can we handle the eop buffer as part of the 
kernel metadata for compute user queues rather than having the user

specify it?


Sure, we can do it.

- Shashank



Alex


  struct drm_amdgpu_userq_in {
--
2.43.2

Re: [PATCH v9 12/14] drm/amdgpu: enable SDMA usermode queues

[Sharma, Shashank]

On 01/05/2024 22:41, Alex Deucher wrote:

On Fri, Apr 26, 2024 at 10:27 AM Shashank Sharma
 wrote:

This patch does necessary modifications to enable the SDMA
usermode queues using the existing userqueue infrastructure.

V9: introduced this patch in the series

Cc: Christian König 
Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
Signed-off-by: Arvind Yadav 
Signed-off-by: Srinivasan Shanmugam 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c| 2 +-
  drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c | 4 
  drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c   | 3 +++
  3 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
index 781283753804..e516487e8db9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
@@ -189,7 +189,7 @@ amdgpu_userqueue_create(struct drm_file *filp, union 
drm_amdgpu_userq *args)
 int qid, r = 0;

 /* Usermode queues are only supported for GFX/SDMA engines as of now */
-   if (args->in.ip_type != AMDGPU_HW_IP_GFX) {
+   if (args->in.ip_type != AMDGPU_HW_IP_GFX && args->in.ip_type != 
AMDGPU_HW_IP_DMA) {
 DRM_ERROR("Usermode queue doesn't support IP type %u\n", 
args->in.ip_type);
 return -EINVAL;
 }
diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
index a6c3037d2d1f..a5e270eda37b 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
@@ -182,6 +182,10 @@ static int mes_v11_0_userq_create_ctx_space(struct 
amdgpu_userq_mgr *uq_mgr,
 return r;
 }

+   /* We don't need to set other FW objects for SDMA queues */
+   if (queue->queue_type == AMDGPU_HW_IP_DMA)
+   return 0;
+
 /* Shadow and GDS objects come directly from userspace */
 mqd->shadow_base_lo = mqd_user->shadow_va & 0xFFFC;
 mqd->shadow_base_hi = upper_32_bits(mqd_user->shadow_va);
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c 
b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
index 361835a61f2e..90354a70c807 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
@@ -1225,6 +1225,8 @@ static int sdma_v6_0_early_init(void *handle)
 return 0;
  }

+extern const struct amdgpu_userq_funcs userq_mes_v11_0_funcs;

Can you include the header rather than adding an extern?

Noted,



+
  static int sdma_v6_0_sw_init(void *handle)
  {
 struct amdgpu_ring *ring;
@@ -1265,6 +1267,7 @@ static int sdma_v6_0_sw_init(void *handle)
 return -EINVAL;
 }

+   adev->userq_funcs[AMDGPU_HW_IP_DMA] = _mes_v11_0_funcs;
 return r;
  }

I think we need a new mqd descriptor in amdgpu_drm.h as well since the
sdma metadata is different from gfx and compute.


Can you please elaborate on this ? AFAIK SDMA queue doesn't need any 
specific metadata objects (like GFX).


- Shashank


Alex


--
2.43.2

Re: [PATCH v9 08/14] drm/amdgpu: map wptr BO into GART

[Sharma, Shashank]

On 01/05/2024 23:36, Alex Deucher wrote:

On Fri, Apr 26, 2024 at 9:57 AM Shashank Sharma  wrote:

To support oversubscription, MES FW expects WPTR BOs to
be mapped into GART, before they are submitted to usermode
queues. This patch adds a function for the same.

V4: fix the wptr value before mapping lookup (Bas, Christian).

V5: Addressed review comments from Christian:
 - Either pin object or allocate from GART, but not both.
 - All the handling must be done with the VM locks held.

V7: Addressed review comments from Christian:
 - Do not take vm->eviction_lock
 - Use amdgpu_bo_gpu_offset to get the wptr_bo GPU offset

V8: Rebase
V9: Changed the function names from gfx_v11* to mes_v11*

Cc: Alex Deucher 
Cc: Christian Koenig 
Signed-off-by: Shashank Sharma 
Signed-off-by: Arvind Yadav 
---
  .../gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c  | 77 +++
  .../gpu/drm/amd/include/amdgpu_userqueue.h|  1 +
  2 files changed, 78 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
index 8d2cd61af26b..37b80626e792 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
@@ -30,6 +30,74 @@
  #define AMDGPU_USERQ_PROC_CTX_SZ PAGE_SIZE
  #define AMDGPU_USERQ_GANG_CTX_SZ PAGE_SIZE

+static int
+mes_v11_0_map_gtt_bo_to_gart(struct amdgpu_device *adev, struct amdgpu_bo *bo)
+{
+   int ret;
+
+   ret = amdgpu_bo_reserve(bo, true);
+   if (ret) {
+   DRM_ERROR("Failed to reserve bo. ret %d\n", ret);
+   goto err_reserve_bo_failed;
+   }
+
+   ret = amdgpu_ttm_alloc_gart(>tbo);
+   if (ret) {
+   DRM_ERROR("Failed to bind bo to GART. ret %d\n", ret);
+   goto err_map_bo_gart_failed;
+   }
+
+   amdgpu_bo_unreserve(bo);
+   bo = amdgpu_bo_ref(bo);
+
+   return 0;
+
+err_map_bo_gart_failed:
+   amdgpu_bo_unreserve(bo);
+err_reserve_bo_failed:
+   return ret;
+}
+
+static int
+mes_v11_0_create_wptr_mapping(struct amdgpu_userq_mgr *uq_mgr,
+ struct amdgpu_usermode_queue *queue,
+ uint64_t wptr)
+{
+   struct amdgpu_device *adev = uq_mgr->adev;
+   struct amdgpu_bo_va_mapping *wptr_mapping;
+   struct amdgpu_vm *wptr_vm;
+   struct amdgpu_userq_obj *wptr_obj = >wptr_obj;
+   int ret;
+
+   wptr_vm = queue->vm;
+   ret = amdgpu_bo_reserve(wptr_vm->root.bo, false);
+   if (ret)
+   return ret;
+
+   wptr &= AMDGPU_GMC_HOLE_MASK;
+   wptr_mapping = amdgpu_vm_bo_lookup_mapping(wptr_vm, wptr >> PAGE_SHIFT);
+   amdgpu_bo_unreserve(wptr_vm->root.bo);
+   if (!wptr_mapping) {
+   DRM_ERROR("Failed to lookup wptr bo\n");
+   return -EINVAL;
+   }
+
+   wptr_obj->obj = wptr_mapping->bo_va->base.bo;
+   if (wptr_obj->obj->tbo.base.size > PAGE_SIZE) {
+   DRM_ERROR("Requested GART mapping for wptr bo larger than one 
page\n");
+   return -EINVAL;
+   }
+
+   ret = mes_v11_0_map_gtt_bo_to_gart(adev, wptr_obj->obj);
+   if (ret) {
+   DRM_ERROR("Failed to map wptr bo to GART\n");
+   return ret;
+   }
+
+   queue->wptr_obj.gpu_addr = amdgpu_bo_gpu_offset_no_check(wptr_obj->obj);

The wptr virtual address from the user may not be at offset 0 from the
start of the object.  We should add the offset to the base vmid0 GPU
address.


can you please elaborate a bit here ? wptr_obj->obj is already mapped to 
gart, do we still need this ?


- Shashank



Alex


+   return 0;
+}
+
  static int mes_v11_0_userq_map(struct amdgpu_userq_mgr *uq_mgr,
struct amdgpu_usermode_queue *queue,
struct amdgpu_mqd_prop *userq_props)
@@ -61,6 +129,7 @@ static int mes_v11_0_userq_map(struct amdgpu_userq_mgr 
*uq_mgr,
 queue_input.queue_size = userq_props->queue_size >> 2;
 queue_input.doorbell_offset = userq_props->doorbell_index;
 queue_input.page_table_base_addr = 
amdgpu_gmc_pd_addr(queue->vm->root.bo);
+   queue_input.wptr_mc_addr = queue->wptr_obj.gpu_addr;

 amdgpu_mes_lock(>mes);
 r = adev->mes.funcs->add_hw_queue(>mes, _input);
@@ -187,6 +256,13 @@ static int mes_v11_0_userq_mqd_create(struct 
amdgpu_userq_mgr *uq_mgr,
 goto free_mqd;
 }

+   /* FW expects WPTR BOs to be mapped into GART */
+   r = mes_v11_0_create_wptr_mapping(uq_mgr, queue, 
userq_props->wptr_gpu_addr);
+   if (r) {
+   DRM_ERROR("Failed to create WPTR mapping\n");
+   goto free_ctx;
+   }
+
 /* Map userqueue into FW using MES */
 r = mes_v11_0_userq_map(uq_mgr, queue, userq_props);
 if (r) {
@@ -216,6 +292,7 @@ mes_v11_0_userq_mqd_destroy(struct amdgpu_userq_mgr *uq_mgr,
 struct 

Re: [PATCH v9 06/14] drm/amdgpu: create context space for usermode queue

[Sharma, Shashank]

On 01/05/2024 23:11, Alex Deucher wrote:

On Fri, Apr 26, 2024 at 10:07 AM Shashank Sharma
 wrote:

The FW expects us to allocate at least one page as context
space to process gang, process, GDS and FW  related work.
This patch creates a joint object for the same, and calculates
GPU space offsets of these spaces.

V1: Addressed review comments on RFC patch:
 Alex: Make this function IP specific

V2: Addressed review comments from Christian
 - Allocate only one object for total FW space, and calculate
   offsets for each of these objects.

V3: Integration with doorbell manager

V4: Review comments:
 - Remove shadow from FW space list from cover letter (Alex)
 - Alignment of macro (Luben)

V5: Merged patches 5 and 6 into this single patch
 Addressed review comments:
 - Use lower_32_bits instead of mask (Christian)
 - gfx_v11_0 instead of gfx_v11 in function names (Alex)
 - Shadow and GDS objects are now coming from userspace (Christian,
   Alex)

V6:
 - Add a comment to replace amdgpu_bo_create_kernel() with
   amdgpu_bo_create() during fw_ctx object creation (Christian).
 - Move proc_ctx_gpu_addr, gang_ctx_gpu_addr and fw_ctx_gpu_addr out
   of generic queue structure and make it gen11 specific (Alex).

V7:
- Using helper function to create/destroy userqueue objects.
- Removed FW object space allocation.

V8:
- Updating FW object address from user values.

V9:
- uppdated function name from gfx_v11_* to mes_v11_*

Cc: Alex Deucher 
Cc: Christian Koenig 
Signed-off-by: Shashank Sharma 
Signed-off-by: Arvind Yadav 
---
  .../gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c  | 43 +++
  .../gpu/drm/amd/include/amdgpu_userqueue.h|  1 +
  2 files changed, 44 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
index 9e7dee77d344..9f9fdcb9c294 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
@@ -27,6 +27,41 @@
  #include "mes_v11_0.h"
  #include "amdgpu_userqueue.h"

+#define AMDGPU_USERQ_PROC_CTX_SZ PAGE_SIZE
+#define AMDGPU_USERQ_GANG_CTX_SZ PAGE_SIZE
+
+static int mes_v11_0_userq_create_ctx_space(struct amdgpu_userq_mgr *uq_mgr,
+   struct amdgpu_usermode_queue *queue,
+   struct drm_amdgpu_userq_mqd 
*mqd_user)
+{
+   struct amdgpu_userq_obj *ctx = >fw_obj;
+   struct v11_gfx_mqd *mqd = queue->mqd.cpu_ptr;
+   int r, size;
+
+   /*
+* The FW expects at least one page space allocated for
+* process ctx and gang ctx each. Create an object
+* for the same.
+*/
+   size = AMDGPU_USERQ_PROC_CTX_SZ + AMDGPU_USERQ_GANG_CTX_SZ;
+   r = amdgpu_userqueue_create_object(uq_mgr, ctx, size);

Is this per queue or per context?  I.e., is this shared with all
queues associated with an fd?


This is per queue object, required for MES mapping of a queue.

- Shashank


Alex


+   if (r) {
+   DRM_ERROR("Failed to allocate ctx space bo for userqueue, 
err:%d\n", r);
+   return r;
+   }
+
+   /* Shadow and GDS objects come directly from userspace */
+   mqd->shadow_base_lo = mqd_user->shadow_va & 0xFFFC;
+   mqd->shadow_base_hi = upper_32_bits(mqd_user->shadow_va);
+
+   mqd->gds_bkup_base_lo = mqd_user->gds_va & 0xFFFC;
+   mqd->gds_bkup_base_hi = upper_32_bits(mqd_user->gds_va);
+
+   mqd->fw_work_area_base_lo = mqd_user->csa_va & 0xFFFC;
+   mqd->fw_work_area_base_hi = upper_32_bits(mqd_user->csa_va);
+   return 0;
+}
+
  static int mes_v11_0_userq_mqd_create(struct amdgpu_userq_mgr *uq_mgr,
   struct drm_amdgpu_userq_in *args_in,
   struct amdgpu_usermode_queue *queue)
@@ -82,6 +117,13 @@ static int mes_v11_0_userq_mqd_create(struct 
amdgpu_userq_mgr *uq_mgr,
 goto free_mqd;
 }

+   /* Create BO for FW operations */
+   r = mes_v11_0_userq_create_ctx_space(uq_mgr, queue, mqd_user);
+   if (r) {
+   DRM_ERROR("Failed to allocate BO for userqueue (%d)", r);
+   goto free_mqd;
+   }
+
 return 0;

  free_mqd:
@@ -100,6 +142,7 @@ static void
  mes_v11_0_userq_mqd_destroy(struct amdgpu_userq_mgr *uq_mgr,
 struct amdgpu_usermode_queue *queue)
  {
+   amdgpu_userqueue_destroy_object(uq_mgr, >fw_obj);
 kfree(queue->userq_prop);
 amdgpu_userqueue_destroy_object(uq_mgr, >mqd);
  }
diff --git a/drivers/gpu/drm/amd/include/amdgpu_userqueue.h 
b/drivers/gpu/drm/amd/include/amdgpu_userqueue.h
index bbd29f68b8d4..643f31474bd8 100644
--- a/drivers/gpu/drm/amd/include/amdgpu_userqueue.h
+++ b/drivers/gpu/drm/amd/include/amdgpu_userqueue.h
@@ -44,6 +44,7 @@ struct amdgpu_usermode_queue {
 struct amdgpu_userq_mgr 

Re: [PATCH v9 05/14] drm/amdgpu: create MES-V11 usermode queue for GFX

[Sharma, Shashank]

On 01/05/2024 22:50, Alex Deucher wrote:

On Fri, Apr 26, 2024 at 9:48 AM Shashank Sharma  wrote:

A Memory queue descriptor (MQD) of a userqueue defines it in
the hw's context. As MQD format can vary between different
graphics IPs, we need gfx GEN specific handlers to create MQDs.

This patch:
- Adds a new file which will be used for MES based userqueue
   functions targeting GFX and SDMA IP.
- Introduces MQD handler functions for the usermode queues.
- Adds new functions to create and destroy userqueue MQD for
   MES-V11 for GFX IP.

V1: Worked on review comments from Alex:
 - Make MQD functions GEN and IP specific

V2: Worked on review comments from Alex:
 - Reuse the existing adev->mqd[ip] for MQD creation
 - Formatting and arrangement of code

V3:
 - Integration with doorbell manager

V4: Review comments addressed:
 - Do not create a new file for userq, reuse gfx_v11_0.c (Alex)
 - Align name of structure members (Luben)
 - Don't break up the Cc tag list and the Sob tag list in commit
   message (Luben)
V5:
- No need to reserve the bo for MQD (Christian).
- Some more changes to support IP specific MQD creation.

V6:
- Add a comment reminding us to replace the amdgpu_bo_create_kernel()
  calls while creating MQD object to amdgpu_bo_create() once eviction
  fences are ready (Christian).

V7:
- Re-arrange userqueue functions in adev instead of uq_mgr (Alex)
- Use memdup_user instead of copy_from_user (Christian)

V9:
- Moved userqueue code from gfx_v11_0.c to new file mes_v11_0.c so
  that it can be reused for SDMA userqueues as well (Shashank, Alex)

Cc: Alex Deucher 
Cc: Christian Koenig 
Signed-off-by: Shashank Sharma 
Signed-off-by: Arvind Yadav 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |   3 +-
  drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c|   4 +
  .../gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c  | 110 ++
  3 files changed, 116 insertions(+), 1 deletion(-)
  create mode 100644 drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile
index 05a2d1714070..a640bfa468ad 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -184,7 +184,8 @@ amdgpu-y += \
  amdgpu-y += \
 amdgpu_mes.o \
 mes_v10_1.o \
-   mes_v11_0.o
+   mes_v11_0.o \
+   mes_v11_0_userqueue.o

  # add UVD block
  amdgpu-y += \
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c 
b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
index f7325b02a191..525bd0f4d3f7 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
@@ -1331,6 +1331,8 @@ static int gfx_v11_0_rlc_backdoor_autoload_enable(struct 
amdgpu_device *adev)
 return 0;
  }

+extern const struct amdgpu_userq_funcs userq_mes_v11_0_funcs;
+
  static int gfx_v11_0_sw_init(void *handle)
  {
 int i, j, k, r, ring_id = 0;
@@ -1347,6 +1349,7 @@ static int gfx_v11_0_sw_init(void *handle)
 adev->gfx.mec.num_mec = 2;
 adev->gfx.mec.num_pipe_per_mec = 4;
 adev->gfx.mec.num_queue_per_pipe = 4;
+   adev->userq_funcs[AMDGPU_HW_IP_GFX] = _mes_v11_0_funcs;
 break;
 case IP_VERSION(11, 0, 1):
 case IP_VERSION(11, 0, 4):
@@ -1358,6 +1361,7 @@ static int gfx_v11_0_sw_init(void *handle)
 adev->gfx.mec.num_mec = 1;
 adev->gfx.mec.num_pipe_per_mec = 4;
 adev->gfx.mec.num_queue_per_pipe = 4;
+   adev->userq_funcs[AMDGPU_HW_IP_GFX] = _mes_v11_0_funcs;

Does this work on APUs yet?  If not, we should limit it to just dGPUs
for now.


I think if we get an APU which is GFX_v11 based it must work on it.


  Also, we should add minimum firmware version checks for user
queue support.


Noted.

- Shashank




 break;
 default:
 adev->gfx.me.num_me = 1;
diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
new file mode 100644
index ..9e7dee77d344
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
@@ -0,0 +1,110 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2024 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF 

Re: [PATCH v9 01/14] drm/amdgpu: UAPI for user queue management

[Sharma, Shashank]

Hey Alex,

On 01/05/2024 22:39, Alex Deucher wrote:

On Fri, Apr 26, 2024 at 10:07 AM Shashank Sharma
 wrote:

From: Alex Deucher 

This patch intorduces new UAPI/IOCTL for usermode graphics
queue. The userspace app will fill this structure and request
the graphics driver to add a graphics work queue for it. The
output of this UAPI is a queue id.

This UAPI maps the queue into GPU, so the graphics app can start
submitting work to the queue as soon as the call returns.

V2: Addressed review comments from Alex and Christian
 - Make the doorbell offset's comment clearer
 - Change the output parameter name to queue_id

V3: Integration with doorbell manager

V4:
 - Updated the UAPI doc (Pierre-Eric)
 - Created a Union for engine specific MQDs (Alex)
 - Added Christian's R-B
V5:
 - Add variables for GDS and CSA in MQD structure (Alex)
 - Make MQD data a ptr-size pair instead of union (Alex)

V9:
- renamed struct drm_amdgpu_userq_mqd_gfx_v11 to struct
  drm_amdgpu_userq_mqd as its being used for SDMA and
  compute queues as well

Cc: Alex Deucher 
Cc: Christian Koenig 
Reviewed-by: Christian König 
Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  include/uapi/drm/amdgpu_drm.h | 110 ++
  1 file changed, 110 insertions(+)

diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
index 96e32dafd4f0..22f56a30f7cb 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -54,6 +54,7 @@ extern "C" {
  #define DRM_AMDGPU_VM  0x13
  #define DRM_AMDGPU_FENCE_TO_HANDLE 0x14
  #define DRM_AMDGPU_SCHED   0x15
+#define DRM_AMDGPU_USERQ   0x16

  #define DRM_IOCTL_AMDGPU_GEM_CREATEDRM_IOWR(DRM_COMMAND_BASE + 
DRM_AMDGPU_GEM_CREATE, union drm_amdgpu_gem_create)
  #define DRM_IOCTL_AMDGPU_GEM_MMAP  DRM_IOWR(DRM_COMMAND_BASE + 
DRM_AMDGPU_GEM_MMAP, union drm_amdgpu_gem_mmap)
@@ -71,6 +72,7 @@ extern "C" {
  #define DRM_IOCTL_AMDGPU_VMDRM_IOWR(DRM_COMMAND_BASE + 
DRM_AMDGPU_VM, union drm_amdgpu_vm)
  #define DRM_IOCTL_AMDGPU_FENCE_TO_HANDLE DRM_IOWR(DRM_COMMAND_BASE + 
DRM_AMDGPU_FENCE_TO_HANDLE, union drm_amdgpu_fence_to_handle)
  #define DRM_IOCTL_AMDGPU_SCHED DRM_IOW(DRM_COMMAND_BASE + 
DRM_AMDGPU_SCHED, union drm_amdgpu_sched)
+#define DRM_IOCTL_AMDGPU_USERQ DRM_IOW(DRM_COMMAND_BASE + 
DRM_AMDGPU_USERQ, union drm_amdgpu_userq)

  /**
   * DOC: memory domains
@@ -317,6 +319,114 @@ union drm_amdgpu_ctx {
 union drm_amdgpu_ctx_out out;
  };

+/* user queue IOCTL */
+#define AMDGPU_USERQ_OP_CREATE 1
+#define AMDGPU_USERQ_OP_FREE   2
+
+/* Flag to indicate secure buffer related workload, unused for now */
+#define AMDGPU_USERQ_MQD_FLAGS_SECURE  (1 << 0)
+/* Flag to indicate AQL workload, unused for now */
+#define AMDGPU_USERQ_MQD_FLAGS_AQL (1 << 1)
+
+/*
+ * MQD (memory queue descriptor) is a set of parameters which allow
+ * the GPU to uniquely define and identify a usermode queue. This
+ * structure defines the MQD for GFX-V11 IP ver 0.
+ */
+struct drm_amdgpu_userq_mqd {

Maybe rename this to drm_amdgpu_gfx_userq_mqd since it's gfx specific.
Then we can add different MQDs for SDMA, compute, etc. as they have
different metadata.  E.g., the shadow and CSA are gfx only.



Actually this was named drm_amdgpu_userq_mqd_gfx_v11_0 until the last 
patchset, but then I realized that apart from the objects (gds/shadow 
va) nothing is gfx specific, its actually required for every userqueue 
IP which is MES based, so I thought it would be an overkill to create 
multiple structures for almost the same data. If you feel strong about 
this, I can change it again.


- Shashank


Alex



+   /**
+* @queue_va: Virtual address of the GPU memory which holds the queue
+* object. The queue holds the workload packets.
+*/
+   __u64   queue_va;
+   /**
+* @queue_size: Size of the queue in bytes, this needs to be 256-byte
+* aligned.
+*/
+   __u64   queue_size;
+   /**
+* @rptr_va : Virtual address of the GPU memory which holds the ring 
RPTR.
+* This object must be at least 8 byte in size and aligned to 8-byte 
offset.
+*/
+   __u64   rptr_va;
+   /**
+* @wptr_va : Virtual address of the GPU memory which holds the ring 
WPTR.
+* This object must be at least 8 byte in size and aligned to 8-byte 
offset.
+*
+* Queue, RPTR and WPTR can come from the same object, as long as the 
size
+* and alignment related requirements are met.
+*/
+   __u64   wptr_va;
+   /**
+* @shadow_va: Virtual address of the GPU memory to hold the shadow 
buffer.
+* This must be a from a separate GPU object, and must be at least 
4-page
+* sized.
+*/
+   __u64   shadow_va;
+   /**
+* @gds_va: Virtual address of the GPU memory to hold the GDS buffer.
+

Re: [PATCH] drm/amdgpu: fix MES HQD masks

[Sharma, Shashank]

On 05/04/2024 18:39, Joshi, Mukul wrote:

[AMD Official Use Only - General]


-Original Message-
From: Sharma, Shashank 
Sent: Friday, April 5, 2024 11:37 AM
To: Joshi, Mukul ; amd-gfx@lists.freedesktop.org
Cc: Koenig, Christian ; Deucher, Alexander
; Yadav, Arvind 
Subject: Re: [PATCH] drm/amdgpu: fix MES HQD masks


On 05/04/2024 17:26, Joshi, Mukul wrote:

[AMD Official Use Only - General]


-Original Message-
From: amd-gfx  On Behalf Of
Shashank Sharma
Sent: Friday, April 5, 2024 10:36 AM
To: amd-gfx@lists.freedesktop.org
Cc: Sharma, Shashank ; Koenig, Christian
; Deucher, Alexander
; Yadav, Arvind 
Subject: [PATCH] drm/amdgpu: fix MES HQD masks

Caution: This message originated from an External Source. Use proper
caution when opening attachments, clicking links, or responding.


This patch fixes the existing HQD masks prepared during the MES

initialization.

These existing masks values were causing problems when we try to
enable GFX oversubscription.

Cc: Christian König 
Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
Signed-off-by: Arvind Yadav 
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c |  3 ---
drivers/gpu/drm/amd/amdgpu/mes_v10_1.c  | 15 ++-
drivers/gpu/drm/amd/amdgpu/mes_v11_0.c  | 15 ++-
   3 files changed, 28 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
index da48b6da0107..7db80ffda33f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
@@ -148,9 +148,6 @@ int amdgpu_mes_init(struct amdgpu_device

*adev)

  adev->mes.compute_hqd_mask[i] = 0xc;
  }

-   for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++)
-   adev->mes.gfx_hqd_mask[i] = i ? 0 : 0xfffe;
-
  for (i = 0; i < AMDGPU_MES_MAX_SDMA_PIPES; i++) {
  if (amdgpu_ip_version(adev, SDMA0_HWIP, 0) <
  IP_VERSION(6, 0, 0)) diff --git
a/drivers/gpu/drm/amd/amdgpu/mes_v10_1.c
b/drivers/gpu/drm/amd/amdgpu/mes_v10_1.c
index 1e5ad1e08d2a..9217914f824d 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v10_1.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v10_1.c
@@ -266,6 +266,19 @@ static int mes_v10_1_query_sched_status(struct
amdgpu_mes *mes)
  offsetof(union MESAPI__QUERY_MES_STATUS,
api_status));  }

+static inline uint32_t mes_v10_get_gfx_hqd_mask(int pipe_index) {
+   /* Pipe 1 can't be used for MES due to HW limitation */
+   if (pipe_index == 1)
+   return 0;
+
+   /*
+* GFX V10 supports 2 queues, but we want to keep queue 0
+* reserved for kernel, so enable only queue 1 (1<<1) for MES.
+*/
+   return 0x2;
+}
+
   static int mes_v10_1_set_hw_resources(struct amdgpu_mes *mes)  {
  int i;
@@ -291,7 +304,7 @@ static int mes_v10_1_set_hw_resources(struct
amdgpu_mes *mes)
  mes->compute_hqd_mask[i];

  for (i = 0; i < MAX_GFX_PIPES; i++)
-   mes_set_hw_res_pkt.gfx_hqd_mask[i] = mes->gfx_hqd_mask[i];
+   mes_set_hw_res_pkt.gfx_hqd_mask[i] =
+ mes_v10_get_gfx_hqd_mask(i);

  for (i = 0; i < MAX_SDMA_PIPES; i++)
  mes_set_hw_res_pkt.sdma_hqd_mask[i] =
mes->sdma_hqd_mask[i]; diff --git
a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
index 26d71a22395d..b7dcd936afc8 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
@@ -360,6 +360,19 @@ static int mes_v11_0_misc_op(struct

amdgpu_mes

*mes,
  offsetof(union MESAPI__MISC, api_status));
}

+static inline uint32_t mes_v11_get_gfx_hqd_mask(int pipe_index) {
+   /* Pipe 1 can't be used for MES due to HW limitation */
+   if (pipe_index == 1)
+   return 0;
+
+   /*
+* GFX V10 supports 2 queues, but we want to keep queue 0
+* reserved for kernel, so enable only queue 1 (1<<1) for MES.
+*/
+   return 0x2;
+}

There is no difference between this function and the corresponding function

written for GFX10.

Why not make this a common function instead?

This is deliberate, to indicate that the HQD mask can be different for each GFX
IP version, so as the number of pipes and queue per pipe. Also the limitation
on pipe 1 will not be there for future versions.


But for now this is common for both GFX10 and GFX11. When we have a case where 
this changes in future, we can implement a new
function specific for that GFX IP version. For now, this should be a common 
function.

Regards,
  Mukul


Mukul,

This was already in a common function and this could have been fixed 
there, but the reason to write two function is to make the HQD mask 
setup IP file specific.


There would be some follow up patches for the same for the SDMA IP as 
well. As you can see, there are multiple common/duplicate functions 
already existing 

Re: [PATCH] drm/amdgpu: fix MES HQD masks

[Sharma, Shashank]

On 05/04/2024 17:26, Joshi, Mukul wrote:

[AMD Official Use Only - General]


-Original Message-
From: amd-gfx  On Behalf Of
Shashank Sharma
Sent: Friday, April 5, 2024 10:36 AM
To: amd-gfx@lists.freedesktop.org
Cc: Sharma, Shashank ; Koenig, Christian
; Deucher, Alexander
; Yadav, Arvind 
Subject: [PATCH] drm/amdgpu: fix MES HQD masks

Caution: This message originated from an External Source. Use proper caution
when opening attachments, clicking links, or responding.


This patch fixes the existing HQD masks prepared during the MES initialization.
These existing masks values were causing problems when we try to enable GFX
oversubscription.

Cc: Christian König 
Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
Signed-off-by: Arvind Yadav 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c |  3 ---
drivers/gpu/drm/amd/amdgpu/mes_v10_1.c  | 15 ++-
drivers/gpu/drm/amd/amdgpu/mes_v11_0.c  | 15 ++-
  3 files changed, 28 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
index da48b6da0107..7db80ffda33f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
@@ -148,9 +148,6 @@ int amdgpu_mes_init(struct amdgpu_device *adev)
 adev->mes.compute_hqd_mask[i] = 0xc;
 }

-   for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++)
-   adev->mes.gfx_hqd_mask[i] = i ? 0 : 0xfffe;
-
 for (i = 0; i < AMDGPU_MES_MAX_SDMA_PIPES; i++) {
 if (amdgpu_ip_version(adev, SDMA0_HWIP, 0) <
 IP_VERSION(6, 0, 0)) diff --git
a/drivers/gpu/drm/amd/amdgpu/mes_v10_1.c
b/drivers/gpu/drm/amd/amdgpu/mes_v10_1.c
index 1e5ad1e08d2a..9217914f824d 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v10_1.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v10_1.c
@@ -266,6 +266,19 @@ static int mes_v10_1_query_sched_status(struct
amdgpu_mes *mes)
 offsetof(union MESAPI__QUERY_MES_STATUS, api_status)); 
 }

+static inline uint32_t mes_v10_get_gfx_hqd_mask(int pipe_index) {
+   /* Pipe 1 can't be used for MES due to HW limitation */
+   if (pipe_index == 1)
+   return 0;
+
+   /*
+* GFX V10 supports 2 queues, but we want to keep queue 0
+* reserved for kernel, so enable only queue 1 (1<<1) for MES.
+*/
+   return 0x2;
+}
+
  static int mes_v10_1_set_hw_resources(struct amdgpu_mes *mes)  {
 int i;
@@ -291,7 +304,7 @@ static int mes_v10_1_set_hw_resources(struct
amdgpu_mes *mes)
 mes->compute_hqd_mask[i];

 for (i = 0; i < MAX_GFX_PIPES; i++)
-   mes_set_hw_res_pkt.gfx_hqd_mask[i] = mes->gfx_hqd_mask[i];
+   mes_set_hw_res_pkt.gfx_hqd_mask[i] =
+ mes_v10_get_gfx_hqd_mask(i);

 for (i = 0; i < MAX_SDMA_PIPES; i++)
 mes_set_hw_res_pkt.sdma_hqd_mask[i] = mes->sdma_hqd_mask[i];
diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
index 26d71a22395d..b7dcd936afc8 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
@@ -360,6 +360,19 @@ static int mes_v11_0_misc_op(struct amdgpu_mes
*mes,
 offsetof(union MESAPI__MISC, api_status));  }

+static inline uint32_t mes_v11_get_gfx_hqd_mask(int pipe_index) {
+   /* Pipe 1 can't be used for MES due to HW limitation */
+   if (pipe_index == 1)
+   return 0;
+
+   /*
+* GFX V10 supports 2 queues, but we want to keep queue 0
+* reserved for kernel, so enable only queue 1 (1<<1) for MES.
+*/
+   return 0x2;
+}

There is no difference between this function and the corresponding function 
written for GFX10.
Why not make this a common function instead?


This is deliberate, to indicate that the HQD mask can be different for 
each GFX IP version, so as the number of pipes and queue per pipe. Also 
the limitation on pipe 1 will not be there for future versions.


- Shashank


Regards,
Mukul


+
  static int mes_v11_0_set_hw_resources(struct amdgpu_mes *mes)  {
 int i;
@@ -385,7 +398,7 @@ static int mes_v11_0_set_hw_resources(struct
amdgpu_mes *mes)
 mes->compute_hqd_mask[i];

 for (i = 0; i < MAX_GFX_PIPES; i++)
-   mes_set_hw_res_pkt.gfx_hqd_mask[i] = mes->gfx_hqd_mask[i];
+   mes_set_hw_res_pkt.gfx_hqd_mask[i] =
+ mes_v11_get_gfx_hqd_mask(i);

 for (i = 0; i < MAX_SDMA_PIPES; i++)
 mes_set_hw_res_pkt.sdma_hqd_mask[i] = mes->sdma_hqd_mask[i];
--
2.43.2

Re: [PATCH] drm/amdgpu/mes11: print MES opcodes rather than numbers

[Sharma, Shashank]

Hi Alex,

On 02/04/2024 02:42, Liu, Shaoyun wrote:

[AMD Official Use Only - General]

[AMD Official Use Only - General]

Comments inline

-Original Message-
From: amd-gfx  On Behalf Of Alex Deucher
Sent: Saturday, March 30, 2024 10:01 AM
To: amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander 
Subject: [PATCH] drm/amdgpu/mes11: print MES opcodes rather than numbers

Makes it easier to review the logs when there are MES errors.

Signed-off-by: Alex Deucher 
---
  drivers/gpu/drm/amd/amdgpu/mes_v11_0.c | 65 --
  1 file changed, 61 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c 
b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
index 072c478665ade..73a4bb0f5ba0f 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
@@ -100,19 +100,51 @@ static const struct amdgpu_ring_funcs 
mes_v11_0_ring_funcs = {
 .insert_nop = amdgpu_ring_insert_nop,
  };

+static const char *mes_v11_0_opcodes[] = {
+   "MES_SCH_API_SET_HW_RSRC",
+   "MES_SCH_API_SET_SCHEDULING_CONFIG",
+   "MES_SCH_API_ADD_QUEUE"
+   "MES_SCH_API_REMOVE_QUEUE"
+   "MES_SCH_API_PERFORM_YIELD"
+   "MES_SCH_API_SET_GANG_PRIORITY_LEVEL"
+   "MES_SCH_API_SUSPEND"
+   "MES_SCH_API_RESUME"
+   "MES_SCH_API_RESET"
+   "MES_SCH_API_SET_LOG_BUFFER"
+   "MES_SCH_API_CHANGE_GANG_PRORITY"
+   "MES_SCH_API_QUERY_SCHEDULER_STATUS"
+   "MES_SCH_API_PROGRAM_GDS"
+   "MES_SCH_API_SET_DEBUG_VMID"
+   "MES_SCH_API_MISC"
+   "MES_SCH_API_UPDATE_ROOT_PAGE_TABLE"
+   "MES_SCH_API_AMD_LOG"
+};
+
+static const char *mes_v11_0_misc_opcodes[] = {
+   "MESAPI_MISC__WRITE_REG",
+   "MESAPI_MISC__INV_GART",
+   "MESAPI_MISC__QUERY_STATUS",
+   "MESAPI_MISC__READ_REG",
+   "MESAPI_MISC__WAIT_REG_MEM",
+   "MESAPI_MISC__SET_SHADER_DEBUGGER",
+};
+
  static int mes_v11_0_submit_pkt_and_poll_completion(struct amdgpu_mes *mes,
 void *pkt, int size,
 int api_status_off)
  {
 int ndw = size / 4;
 signed long r;
-   union MESAPI__ADD_QUEUE *x_pkt = pkt;
+   union MESAPI__MISC *x_pkt = pkt;
 struct MES_API_STATUS *api_status;
 struct amdgpu_device *adev = mes->adev;
 struct amdgpu_ring *ring = >ring;
 unsigned long flags;
 signed long timeout = adev->usec_timeout;

+   if (x_pkt->header.opcode >= MES_SCH_API_MAX)
+   return -EINVAL;
+
 if (amdgpu_emu_mode) {
 timeout *= 100;
 } else if (amdgpu_sriov_vf(adev)) {
@@ -135,13 +167,38 @@ static int 
mes_v11_0_submit_pkt_and_poll_completion(struct amdgpu_mes *mes,
 amdgpu_ring_commit(ring);
 spin_unlock_irqrestore(>ring_lock, fl
-   DRM_DEBUG("MES msg=%d was emitted\n", x_pkt->header.opcode);
+   if (x_pkt->header.opcode == MES_SCH_API_MISC) {
+   if (x_pkt->opcode <= ARRAY_SIZE(mes_v11_0_misc_opcodes))
+   dev_err(adev->dev, "MES msg=%s (%s) was emitted\n",

[shaoyunl]  Shouldn't  we  use DRM_DEBUG  for valid  condition ?

Regards
Shaoyun.liu

+   mes_v11_0_opcodes[x_pkt->header.opcode],
+   mes_v11_0_misc_opcodes[x_pkt->opcode]);
+   else
+   dev_err(adev->dev, "MES msg=%s (%d) was emitted\n",
+   mes_v11_0_opcodes[x_pkt->header.opcode],
+   x_pkt->opcode);
+   } else if (x_pkt->header.opcode < ARRAY_SIZE(mes_v11_0_opcodes))
+   dev_err(adev->dev, "MES msg=%s was emitted\n",
+   mes_v11_0_opcodes[x_pkt->header.opcode]);
+   else
+   dev_err(adev->dev, "MES msg=%d was emitted\n", 
x_pkt->header.opcode);

 r = amdgpu_fence_wait_polling(ring, ring->fence_drv.sync_seq,
   timeout);
 if (r < 1) {
-   DRM_ERROR("MES failed to response msg=%d\n",
- x_pkt->header.opcode);
+   if (x_pkt->header.opcode == MES_SCH_API_MISC) {
+   if (x_pkt->opcode <= ARRAY_SIZE(mes_v11_0_misc_opcodes))
+   dev_err(adev->dev, "MES failed to response msg=%s 
(%s)\n",
+   mes_v11_0_opcodes[x_pkt->header.opcode],
+   mes_v11_0_misc_opcodes[x_pkt->opcode]);
+   else
+   dev_err(adev->dev, "MES failed to response msg=%s 
(%d)\n",
+   mes_v11_0_opcodes[x_pkt->header.opcode], 
x_pkt->opcode);
+   } else if (x_pkt->header.opcode < ARRAY_SIZE(mes_v11_0_opcodes))
+   dev_err(adev->dev, "MES failed to response msg=%s\n",
+   mes_v11_0_opcodes[x_pkt->header.opcode]);
+   

Re: [PATCH] drm/amdgpu: add post reset IP callback

[Sharma, Shashank]

On 03/04/2024 09:31, Yu, Lang wrote:

[AMD Official Use Only - General]


-Original Message-
From: Sharma, Shashank 
Sent: Wednesday, April 3, 2024 3:19 PM
To: Yu, Lang ; Christian König
; amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander ; Koenig, Christian

Subject: Re: [PATCH] drm/amdgpu: add post reset IP callback

Hey Lang,

On 03/04/2024 08:51, Yu, Lang wrote:

[AMD Official Use Only - General]


-Original Message-
From: Christian König 
Sent: Tuesday, April 2, 2024 9:38 PM
To: Yu, Lang ; amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander ; Koenig, Christian
; Sharma, Shashank

Subject: Re: [PATCH] drm/amdgpu: add post reset IP callback

Am 28.03.24 um 05:40 schrieb Lang Yu:

There are use cases which need full GPU functionality (e.g., VM
update, TLB inavildate) when doing a GPU reset.

Especially, the mes/umsch self tests which help validate the hw
state after hw init like ring/ib tests.

I noted that before but just to repeat it once more: We can't do any
MES or UMSCH validation while doing a GPU reset!

Yes, we can just easily disable it if it doesn't work well.
But it doesn't take too much effort to make it work.
It can expose issues as soon as possible and is useful for debugging

purpose.
IMO, its not that useful for debugging as well. In case of a problem, It will 
just
timeout waiting for MES packet write and we will still have to find out the
actual problem which caused MES to go into bad state in the last GPU reset.

The ring and IB tests use some pre-allocated memory we put aside for
the task during driver load and so can execute during GPU reset as well.

If user space can create a VM and allocate memory during GPU reset, it
makes no sense to prevent kernel space from doing that.

I think the objection here is mostly about why to do it at all, when it is not
helpful. It would be just a maintenance overhead.

If you think it is not helpful,  why doing ring/ib tests?
That's because they add value during the bootup, when we know that the 
HW is being initialized in a controlled/systematic way. But in case of a 
GPU reset, it could go anyway and could be neither of those.

I don't think such sanity test is not helpful.

I only talk UMSCH test(different with MES test) here,


Your comment above about ring tests talks about both MES/UMSCH self 
tests, so it attracts attention on both.


If its only about UMSCH tests, I will let someone else to comment on that.

- Shashank


I don't think it has a maintenance overhead.

Regards,
Lang


- Shashank


But for the MES/UMSCH we need a full blown environment with VM and
submission infrastructure and setting that up isn't possible here.

At least for UMSCH test, it only uses VM mapping functionality (if we
can create a VM with cpu update mode, that's enough), it doesn't use
other submission functionality.
It is actually a compute context.


Regards,
Lang


Adding Shashank as well, but I think we should probably just
completely remove those from the kernel.

Regards,
Christian.


Add a post reset IP callback to handle such use cases which will be
executed after GPU reset succeeds.

Signed-off-by: Lang Yu 
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 24

++

drivers/gpu/drm/amd/include/amd_shared.h   |  3 +++
2 files changed, 27 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 12dc71a6b5db..feeab9397aab 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -5556,6 +5556,27 @@ static int

amdgpu_device_health_check(struct

list_head *device_list_handle)

   return ret;
}

+static int amdgpu_device_ip_post_reset(struct amdgpu_device *adev) {
+uint32_t i;
+int r;
+
+for (i = 0; i < adev->num_ip_blocks; i++) {
+if (!adev->ip_blocks[i].status.valid ||
+!adev->ip_blocks[i].version->funcs->post_reset)
+continue;
+
+r = adev->ip_blocks[i].version->funcs->post_reset(adev);
+if (r) {
+DRM_ERROR("post reset of IP block <%s>

failed %d\n",

+  adev->ip_blocks[i].version->funcs->name, r);
+return r;
+}
+}
+
+return r;
+}
+
/**
 * amdgpu_device_gpu_recover - reset the asic and recover scheduler
 *
@@ -5805,6 +5826,9 @@ int amdgpu_device_gpu_recover(struct

amdgpu_device *adev,

   amdgpu_put_xgmi_hive(hive);
   }

+if (!r && !job_signaled)
+r = amdgpu_device_ip_post_reset(adev);
+
   if (r)
   dev_info(adev->dev, "GPU reset end with ret = %d\n",
r);

diff --git a/drivers/gpu/drm/amd/include/amd_shared.h
b/drivers/gpu/drm/amd/include/amd_shared.h
index b0a6256e89f4..33ce30a8e3ab 100644
--- a/drivers/gpu/drm/amd/include/amd_shared.h
+++ b/drivers/gpu/drm/amd/include/amd_share

Re: [PATCH] drm/amdgpu: add post reset IP callback

[Sharma, Shashank]

Hey Lang,

On 03/04/2024 08:51, Yu, Lang wrote:

[AMD Official Use Only - General]


-Original Message-
From: Christian König 
Sent: Tuesday, April 2, 2024 9:38 PM
To: Yu, Lang ; amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander ; Koenig, Christian
; Sharma, Shashank

Subject: Re: [PATCH] drm/amdgpu: add post reset IP callback

Am 28.03.24 um 05:40 schrieb Lang Yu:

There are use cases which need full GPU functionality (e.g., VM
update, TLB inavildate) when doing a GPU reset.

Especially, the mes/umsch self tests which help validate the hw state
after hw init like ring/ib tests.

I noted that before but just to repeat it once more: We can't do any MES or
UMSCH validation while doing a GPU reset!

Yes, we can just easily disable it if it doesn't work well.
But it doesn't take too much effort to make it work.
It can expose issues as soon as possible and is useful for debugging purpose.
IMO, its not that useful for debugging as well. In case of a problem, It 
will just timeout waiting for MES packet write and we will still have to 
find out the actual problem which caused MES to go into bad state in the 
last GPU reset.



The ring and IB tests use some pre-allocated memory we put aside for the
task during driver load and so can execute during GPU reset as well.

If user space can create a VM and allocate memory during GPU reset,
it makes no sense to prevent kernel space from doing that.


I think the objection here is mostly about why to do it at all, when it 
is not helpful. It would be just a maintenance overhead.


- Shashank


But for the MES/UMSCH we need a full blown environment with VM and
submission infrastructure and setting that up isn't possible here.

At least for UMSCH test, it only uses VM mapping functionality
(if we can create a VM with cpu update mode, that's enough),
it doesn't use other submission functionality.
It is actually a compute context.


Regards,
Lang


Adding Shashank as well, but I think we should probably just completely
remove those from the kernel.

Regards,
Christian.


Add a post reset IP callback to handle such use cases which will be
executed after GPU reset succeeds.

Signed-off-by: Lang Yu 
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 24

++

   drivers/gpu/drm/amd/include/amd_shared.h   |  3 +++
   2 files changed, 27 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 12dc71a6b5db..feeab9397aab 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -5556,6 +5556,27 @@ static int amdgpu_device_health_check(struct

list_head *device_list_handle)

  return ret;
   }

+static int amdgpu_device_ip_post_reset(struct amdgpu_device *adev) {
+uint32_t i;
+int r;
+
+for (i = 0; i < adev->num_ip_blocks; i++) {
+if (!adev->ip_blocks[i].status.valid ||
+!adev->ip_blocks[i].version->funcs->post_reset)
+continue;
+
+r = adev->ip_blocks[i].version->funcs->post_reset(adev);
+if (r) {
+DRM_ERROR("post reset of IP block <%s>

failed %d\n",

+  adev->ip_blocks[i].version->funcs->name, r);
+return r;
+}
+}
+
+return r;
+}
+
   /**
* amdgpu_device_gpu_recover - reset the asic and recover scheduler
*
@@ -5805,6 +5826,9 @@ int amdgpu_device_gpu_recover(struct

amdgpu_device *adev,

  amdgpu_put_xgmi_hive(hive);
  }

+if (!r && !job_signaled)
+r = amdgpu_device_ip_post_reset(adev);
+
  if (r)
  dev_info(adev->dev, "GPU reset end with ret = %d\n", r);

diff --git a/drivers/gpu/drm/amd/include/amd_shared.h
b/drivers/gpu/drm/amd/include/amd_shared.h
index b0a6256e89f4..33ce30a8e3ab 100644
--- a/drivers/gpu/drm/amd/include/amd_shared.h
+++ b/drivers/gpu/drm/amd/include/amd_shared.h
@@ -287,6 +287,7 @@ enum amd_dpm_forced_level;
* @pre_soft_reset: pre soft reset the IP block
* @soft_reset: soft reset the IP block
* @post_soft_reset: post soft reset the IP block
+ * @post_reset: handles IP specific post reset stuff(e.g., self test)
* @set_clockgating_state: enable/disable cg for the IP block
* @set_powergating_state: enable/disable pg for the IP block
* @get_clockgating_state: get current clockgating status @@ -316,11
+317,13 @@ struct amd_ip_funcs {
  int (*pre_soft_reset)(void *handle);
  int (*soft_reset)(void *handle);
  int (*post_soft_reset)(void *handle);
+int (*post_reset)(void *handle);
  int (*set_clockgating_state)(void *handle,
   enum amd_clockgating_state state);
  int (*set_powergating_state)(void *handle,
   enum amd_powergating_state state);
  void (*get_clockgating_state)(void *handle, u64 *flags);
+
   };

Re: [PATCH] drm/amdgpu: fix deadlock while reading mqd from debugfs

[Sharma, Shashank]

Thanks for the patch,

Patch pushed for staging.


Regards

Shashank

On 25/03/2024 00:23, Alex Deucher wrote:

On Sat, Mar 23, 2024 at 4:47 PM Sharma, Shashank
 wrote:


On 23/03/2024 15:52, Johannes Weiner wrote:

On Thu, Mar 14, 2024 at 01:09:57PM -0400, Johannes Weiner wrote:

Hello,

On Fri, Mar 08, 2024 at 12:32:33PM +0100, Christian König wrote:

Am 07.03.24 um 23:07 schrieb Johannes Weiner:

Lastly I went with an open loop instead of a memcpy() as I wasn't
sure if that memory is safe to address a byte at at time.

Shashank pointed out to me in private that byte access would indeed be
safe. However, after actually trying it it won't work because memcpy()
doesn't play nice with mqd being volatile:

/home/hannes/src/linux/linux/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c: In 
function 'amdgpu_debugfs_mqd_read':
/home/hannes/src/linux/linux/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c:550:22: 
warning: passing argument 1 of '__builtin_dynamic_object_size' discards 
'volatil' qualifier from pointer target type [-Wdiscarded-qualifiers]
550 | memcpy(kbuf, mqd, ring->mqd_size);

So I would propose leaving the patch as-is. Shashank, does that sound
good to you?

Friendly ping :)

Shashank, is your Reviewed-by still good for this patch, given the
above?

Ah, sorry I missed this due to some parallel work, and just realized the
memcpy/volatile limitation.

I also feel the need of protecting MQD read under a lock to avoid
parallel change in MQD while we do byte-by-byte copy, but I will add
that in my to-do list.

Please feel free to use my R-b.

Shashank, if the patch looks good, can you pick it up and apply it?

Alex



- Shashank


Thanks

Re: [PATCH] drm/amdgpu : Increase the mes log buffer size as per new MES FW version

[Sharma, Shashank]

[AMD Official Use Only - General]




From: amd-gfx  on behalf of Liu, Shaoyun 

Sent: Monday, March 25, 2024 1:58 PM
To: amd-gfx@lists.freedesktop.org 
Subject: Re: [PATCH] drm/amdgpu : Increase the mes log buffer size as per new 
MES FW version


[AMD Official Use Only - General]


[AMD Official Use Only - General]

Ping

Get Outlook for iOS

From: Liu, Shaoyun 
Sent: Friday, March 22, 2024 12:49:56 PM
To: amd-gfx@lists.freedesktop.org 
Cc: Liu, Shaoyun 
Subject: [PATCH] drm/amdgpu : Increase the mes log buffer size as per new MES 
FW version

>From MES version 0x54, the log entry increased and require the log buffer
size to be increased. The 16k is maximum size agreed

Signed-off-by: shaoyunl 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 5 ++---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h | 1 +
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
index 9ace848e174c..78e4f88f5134 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
@@ -103,7 +103,7 @@ static int amdgpu_mes_event_log_init(struct amdgpu_device 
*adev)
 if (!amdgpu_mes_log_enable)
 return 0;

-   r = amdgpu_bo_create_kernel(adev, PAGE_SIZE, PAGE_SIZE,
+   r = amdgpu_bo_create_kernel(adev, AMDGPU_MES_LOG_BUFFER_SIZE, PAGE_SIZE,

shouldn't this be under a version check >= 0x54 instead of directly changing it 
?

 AMDGPU_GEM_DOMAIN_GTT,
 >mes.event_log_gpu_obj,
 >mes.event_log_gpu_addr,
@@ -1548,12 +1548,11 @@ static int amdgpu_debugfs_mes_event_log_show(struct 
seq_file *m, void *unused)
 uint32_t *mem = (uint32_t *)(adev->mes.event_log_cpu_addr);

 seq_hex_dump(m, "", DUMP_PREFIX_OFFSET, 32, 4,
-mem, PAGE_SIZE, false);
+mem, AMDGPU_MES_LOG_BUFFER_SIZE, false);

 return 0;
 }

-
 DEFINE_SHOW_ATTRIBUTE(amdgpu_debugfs_mes_event_log);

 #endif
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h
index 7d4f93fea937..4c8fc3117ef8 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h
@@ -52,6 +52,7 @@ enum amdgpu_mes_priority_level {

 #define AMDGPU_MES_PROC_CTX_SIZE 0x1000 /* one page area */
 #define AMDGPU_MES_GANG_CTX_SIZE 0x1000 /* one page area */
+#define AMDGPU_MES_LOG_BUFFER_SIZE 0x4000 /* Maximu log buffer size for MES */

Apart from that, it looks good to me.


  *   Shashank

 struct amdgpu_mes_funcs;

--
2.34.1

Re: [PATCH] drm/amdgpu: fix deadlock while reading mqd from debugfs

[Sharma, Shashank]

[AMD Official Use Only - General]

Hey Alex,
Sure, I will pick it up and push it to staging.

Regards
Shashank

From: Alex Deucher 
Sent: Monday, March 25, 2024 12:23 AM
To: Sharma, Shashank 
Cc: Johannes Weiner ; Christian König 
; Deucher, Alexander 
; Koenig, Christian ; 
amd-gfx@lists.freedesktop.org ; 
dri-de...@lists.freedesktop.org ; 
linux-ker...@vger.kernel.org 
Subject: Re: [PATCH] drm/amdgpu: fix deadlock while reading mqd from debugfs

On Sat, Mar 23, 2024 at 4:47 PM Sharma, Shashank
 wrote:
>
>
> On 23/03/2024 15:52, Johannes Weiner wrote:
> > On Thu, Mar 14, 2024 at 01:09:57PM -0400, Johannes Weiner wrote:
> >> Hello,
> >>
> >> On Fri, Mar 08, 2024 at 12:32:33PM +0100, Christian König wrote:
> >>> Am 07.03.24 um 23:07 schrieb Johannes Weiner:
> >>>> Lastly I went with an open loop instead of a memcpy() as I wasn't
> >>>> sure if that memory is safe to address a byte at at time.
> >> Shashank pointed out to me in private that byte access would indeed be
> >> safe. However, after actually trying it it won't work because memcpy()
> >> doesn't play nice with mqd being volatile:
> >>
> >> /home/hannes/src/linux/linux/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c: In 
> >> function 'amdgpu_debugfs_mqd_read':
> >> /home/hannes/src/linux/linux/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c:550:22:
> >>  warning: passing argument 1 of '__builtin_dynamic_object_size' discards 
> >> 'volatil' qualifier from pointer target type [-Wdiscarded-qualifiers]
> >>550 | memcpy(kbuf, mqd, ring->mqd_size);
> >>
> >> So I would propose leaving the patch as-is. Shashank, does that sound
> >> good to you?
> > Friendly ping :)
> >
> > Shashank, is your Reviewed-by still good for this patch, given the
> > above?
>
> Ah, sorry I missed this due to some parallel work, and just realized the
> memcpy/volatile limitation.
>
> I also feel the need of protecting MQD read under a lock to avoid
> parallel change in MQD while we do byte-by-byte copy, but I will add
> that in my to-do list.
>
> Please feel free to use my R-b.

Shashank, if the patch looks good, can you pick it up and apply it?

Alex


>
> - Shashank
>
> > Thanks

Re: [PATCH] drm/amdgpu: fix deadlock while reading mqd from debugfs

[Sharma, Shashank]

On 23/03/2024 15:52, Johannes Weiner wrote:

On Thu, Mar 14, 2024 at 01:09:57PM -0400, Johannes Weiner wrote:

Hello,

On Fri, Mar 08, 2024 at 12:32:33PM +0100, Christian König wrote:

Am 07.03.24 um 23:07 schrieb Johannes Weiner:

Lastly I went with an open loop instead of a memcpy() as I wasn't
sure if that memory is safe to address a byte at at time.

Shashank pointed out to me in private that byte access would indeed be
safe. However, after actually trying it it won't work because memcpy()
doesn't play nice with mqd being volatile:

/home/hannes/src/linux/linux/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c: In 
function 'amdgpu_debugfs_mqd_read':
/home/hannes/src/linux/linux/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c:550:22: 
warning: passing argument 1 of '__builtin_dynamic_object_size' discards 
'volatil' qualifier from pointer target type [-Wdiscarded-qualifiers]
   550 | memcpy(kbuf, mqd, ring->mqd_size);

So I would propose leaving the patch as-is. Shashank, does that sound
good to you?

Friendly ping :)

Shashank, is your Reviewed-by still good for this patch, given the
above?


Ah, sorry I missed this due to some parallel work, and just realized the 
memcpy/volatile limitation.


I also feel the need of protecting MQD read under a lock to avoid 
parallel change in MQD while we do byte-by-byte copy, but I will add 
that in my to-do list.


Please feel free to use my R-b.

- Shashank


Thanks

Re: [PATCH v8] drm/amdgpu: sync page table freeing with tlb flush

[Sharma, Shashank]

On 18/03/2024 16:24, Christian König wrote:

Am 18.03.24 um 16:22 schrieb Sharma, Shashank:


On 18/03/2024 16:01, Christian König wrote:

Am 18.03.24 um 15:44 schrieb Shashank Sharma:

The idea behind this patch is to delay the freeing of PT entry objects
until the TLB flush is done.

This patch:
- Adds a tlb_flush_waitlist in amdgpu_vm_update_params which will 
keep the

   objects that need to be freed after tlb_flush.
- Adds PT entries in this list in amdgpu_vm_ptes_update after finding
   the PT entry.
- Changes functionality of amdgpu_vm_pt_free_dfs from (df_search + 
free)

   to simply freeing of the BOs, also renames it to
   amdgpu_vm_pt_free_list to reflect this same.
- Exports function amdgpu_vm_pt_free_list to be called directly.
- Calls amdgpu_vm_pt_free_list directly from amdgpu_vm_update_range.

V2: rebase
V4: Addressed review comments from Christian
 - add only locked PTEs entries in TLB flush waitlist.
 - do not create a separate function for list flush.
 - do not create a new lock for TLB flush.
 - there is no need to wait on tlb_flush_fence exclusively.

V5: Addressed review comments from Christian
 - change the amdgpu_vm_pt_free_dfs's functionality to simple 
freeing

   of the objects and rename it.
 - add all the PTE objects in params->tlb_flush_waitlist
 - let amdgpu_vm_pt_free_root handle the freeing of BOs 
independently

 - call amdgpu_vm_pt_free_list directly

V6: Rebase
V7: Rebase
V8: Added a NULL check to fix this backtrace issue:
[  415.351447] BUG: kernel NULL pointer dereference, address: 
0008

[  415.359245] #PF: supervisor write access in kernel mode
[  415.365081] #PF: error_code(0x0002) - not-present page
[  415.370817] PGD 101259067 P4D 101259067 PUD 10125a067 PMD 0
[  415.377140] Oops: 0002 [#1] PREEMPT SMP NOPTI
[  415.382004] CPU: 0 PID: 25481 Comm: test_with_MPI.e Tainted: 
G   OE 5.18.2-mi300-build-140423-ubuntu-22.04+ #24
[  415.394437] Hardware name: AMD Corporation Sh51p/Sh51p, BIOS 
RMO1001AS 02/21/2024

[  415.402797] RIP: 0010:amdgpu_vm_ptes_update+0x6fd/0xa10 [amdgpu]
[  415.409648] Code: 4c 89 ff 4d 8d 66 30 e8 f1 ed ff ff 48 85 db 
74 42 48 39 5d a0 74 40 48 8b 53 20 48 8b 4b 18 48 8d 43 18 48 8d 
75 b0 4c 89 ff <48

89 51 08 48 89 0a 49 8b 56 30 48 89 42 08 48 89 53 18 4c 89 63

[  415.430621] RSP: 0018:c9000401f990 EFLAGS: 00010287
[  415.436456] RAX: 888147bb82f0 RBX: 888147bb82d8 RCX: 

[  415.26] RDX:  RSI: c9000401fa30 RDI: 
888161f8
[  415.452397] RBP: c9000401fa80 R08:  R09: 
c9000401fa00
[  415.460368] R10: 0007f0cc R11: 0007f0c85000 R12: 
c9000401fb20
[  415.468340] R13: 0007f0d0 R14: c9000401faf0 R15: 
888161f8
[  415.476312] FS:  7f132ff89840() 
GS:889f87c0() knlGS:

[  415.485350] CS:  0010 DS:  ES:  CR0: 80050033
[  415.491767] CR2: 0008 CR3: 000161d46003 CR4: 
00770ef0

[  415.499738] PKRU: 5554
[  415.502750] Call Trace:
[  415.505482]  
[  415.507825]  amdgpu_vm_update_range+0x32a/0x880 [amdgpu]
[  415.513869]  amdgpu_vm_clear_freed+0x117/0x250 [amdgpu]
[  415.519814] 
amdgpu_amdkfd_gpuvm_unmap_memory_from_gpu+0x18c/0x250 [amdgpu]

[  415.527729]  kfd_ioctl_unmap_memory_from_gpu+0xed/0x340 [amdgpu]
[  415.534551]  kfd_ioctl+0x3b6/0x510 [amdgpu]

Cc: Christian König 
Cc: Alex Deucher 
Cc: Felix Kuehling 
Cc: Rajneesh Bhardwaj 
Acked-by: Felix Kuehling 
Acked-by: Rajneesh Bhardwaj 
Tested-by: Rajneesh Bhardwaj 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    |  5 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |  7 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c | 58 
+--

  3 files changed, 45 insertions(+), 25 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

index 26f1c3359642..eaa402f99fe0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -977,6 +977,7 @@ int amdgpu_vm_update_range(struct amdgpu_device 
*adev, struct amdgpu_vm *vm,

  params.unlocked = unlocked;
  params.needs_flush = flush_tlb;
  params.allow_override = allow_override;
+    INIT_LIST_HEAD(_flush_waitlist);
    /* Implicitly sync to command submissions in the same VM 
before

   * unmapping. Sync to moving fences before mapping.
@@ -1062,8 +1063,10 @@ int amdgpu_vm_update_range(struct 
amdgpu_device *adev, struct amdgpu_vm *vm,

  if (r)
  goto error_unlock;
  -    if (params.needs_flush)
+    if (params.needs_flush) {
  r = amdgpu_vm_tlb_flush(, fence);
+    amdgpu_vm_pt_free_list(adev, );


This is completed independent of the VM flush and should always be 
called.



+    }
    error_unlock:
  amdgpu_vm_eviction_unlock(vm);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
b/drivers/gpu/drm/amd/

Re: [PATCH v9 1/2] drm/amdgpu: implement TLB flush fence

[Sharma, Shashank]

On 18/03/2024 18:30, Joshi, Mukul wrote:

[AMD Official Use Only - General]


-Original Message-
From: amd-gfx  On Behalf Of
Shashank Sharma
Sent: Monday, March 18, 2024 12:12 PM
To: amd-gfx@lists.freedesktop.org
Cc: Koenig, Christian ; Kuehling, Felix
; Bhardwaj, Rajneesh
; Deucher, Alexander
; Sharma, Shashank

Subject: [PATCH v9 1/2] drm/amdgpu: implement TLB flush fence

Caution: This message originated from an External Source. Use proper caution
when opening attachments, clicking links, or responding.


From: Christian Koenig 

The problem is that when (for example) 4k pages are replaced with a single 2M
page we need to wait for change to be flushed out by invalidating the TLB
before the PT can be freed.

Solve this by moving the TLB flush into a DMA-fence object which can be used
to delay the freeing of the PT BOs until it is signaled.

V2: (Shashank)
 - rebase
 - set dma_fence_error only in case of error
 - add tlb_flush fence only when PT/PD BO is locked (Felix)
 - use vm->pasid when f is NULL (Mukul)

V4: - add a wait for (f->dependency) in tlb_fence_work (Christian)
 - move the misplaced fence_create call to the end (Philip)

V5: - free the f->dependency properly

V6: (Shashank)
 - light code movement, moved all the clean-up in previous patch
 - introduce params.needs_flush and its usage in this patch
 - rebase without TLB HW sequence patch

V7:
- Keep the vm->last_update_fence and tlb_cb code until
  we can fix the HW sequencing (Christian)
- Move all the tlb_fence related code in a separate function so that
  its easier to read and review

V9: Addressed review comments from Christian
 - start PT update only when we have callback memory allocated

Cc: Christian Koenig 
Cc: Felix Kuehling 
Cc: Rajneesh Bhardwaj 
Cc: Alex Deucher 
Acked-by: Felix Kuehling 
Acked-by: Rajneesh Bhardwaj 
Tested-by: Rajneesh Bhardwaj 
Reviewed-by: Shashank Sharma 
Signed-off-by: Christian Koenig 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |   3 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c|  64 +++---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h|   8 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_cpu.c|   4 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_sdma.c   |   4 +
  .../gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c  | 112
++
  7 files changed, 175 insertions(+), 22 deletions(-)  create mode 100644
drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile
b/drivers/gpu/drm/amd/amdgpu/Makefile
index 4536c8ad0e11..f24f11ac3e92 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -70,7 +70,8 @@ amdgpu-y += amdgpu_device.o
amdgpu_doorbell_mgr.o amdgpu_kms.o \
 amdgpu_cs.o amdgpu_bios.o amdgpu_benchmark.o \
 atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
 atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
-   amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o
amdgpu_pll.o \
+   amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o
amdgpu_vm_tlb_fence.o \
+   amdgpu_ib.o amdgpu_pll.o \
 amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
 amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o
amdgpu_virt.o \
 amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \ diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 81fb3465e197..104bf600c85f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -885,6 +885,44 @@ static void amdgpu_vm_tlb_seq_cb(struct
dma_fence *fence,
 kfree(tlb_cb);
  }

+/**
+ * amdgpu_vm_tlb_flush - prepare TLB flush
+ *
+ * @params: parameters for update
+ * @fence: input fence to sync TLB flush with
+ * @tlb_cb: the callback structure
+ *
+ * Increments the tlb sequence to make sure that future CS execute a VM
flush.
+ */
+static void
+amdgpu_vm_tlb_flush(struct amdgpu_vm_update_params *params,
+   struct dma_fence **fence,
+   struct amdgpu_vm_tlb_seq_struct *tlb_cb) {
+   struct amdgpu_vm *vm = params->vm;
+
+   if (!fence || !*fence)
+   return;
+
+   tlb_cb->vm = vm;
+   if (!dma_fence_add_callback(*fence, _cb->cb,
+   amdgpu_vm_tlb_seq_cb)) {
+   dma_fence_put(vm->last_tlb_flush);
+   vm->last_tlb_flush = dma_fence_get(*fence);
+   } else {
+   amdgpu_vm_tlb_seq_cb(NULL, _cb->cb);
+   }
+
+   /* Prepare a TLB flush fence to be attached to PTs */
+   if (!params->unlocked && vm->is_compute_context) {
+   amdgpu_vm_tlb_fence_create(params->adev, vm, fence);
+
+   /* Makes sure no PD/PT is freed before the flush */
+   dma_re

Re: [PATCH v9 1/2] drm/amdgpu: implement TLB flush fence

[Sharma, Shashank]

On 18/03/2024 19:10, Christian König wrote:

Am 18.03.24 um 17:11 schrieb Shashank Sharma:

From: Christian Koenig 

The problem is that when (for example) 4k pages are replaced
with a single 2M page we need to wait for change to be flushed
out by invalidating the TLB before the PT can be freed.

Solve this by moving the TLB flush into a DMA-fence object which
can be used to delay the freeing of the PT BOs until it is signaled.

V2: (Shashank)
 - rebase
 - set dma_fence_error only in case of error
 - add tlb_flush fence only when PT/PD BO is locked (Felix)
 - use vm->pasid when f is NULL (Mukul)

V4: - add a wait for (f->dependency) in tlb_fence_work (Christian)
 - move the misplaced fence_create call to the end (Philip)

V5: - free the f->dependency properly

V6: (Shashank)
 - light code movement, moved all the clean-up in previous patch
 - introduce params.needs_flush and its usage in this patch
 - rebase without TLB HW sequence patch

V7:
    - Keep the vm->last_update_fence and tlb_cb code until
  we can fix the HW sequencing (Christian)
    - Move all the tlb_fence related code in a separate function so that
  its easier to read and review

V9: Addressed review comments from Christian
 - start PT update only when we have callback memory allocated

Cc: Christian Koenig 
Cc: Felix Kuehling 
Cc: Rajneesh Bhardwaj 
Cc: Alex Deucher 
Acked-by: Felix Kuehling 
Acked-by: Rajneesh Bhardwaj 
Tested-by: Rajneesh Bhardwaj 
Reviewed-by: Shashank Sharma 
Signed-off-by: Christian Koenig 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |   3 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    |  64 +++---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |   8 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_cpu.c    |   4 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_sdma.c   |   4 +
  .../gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c  | 112 ++
  7 files changed, 175 insertions(+), 22 deletions(-)
  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index 4536c8ad0e11..f24f11ac3e92 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -70,7 +70,8 @@ amdgpu-y += amdgpu_device.o amdgpu_doorbell_mgr.o 
amdgpu_kms.o \

  amdgpu_cs.o amdgpu_bios.o amdgpu_benchmark.o \
  atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
  atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
-    amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o 
amdgpu_pll.o \

+    amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_vm_tlb_fence.o \
+    amdgpu_ib.o amdgpu_pll.o \
  amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
  amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o 
amdgpu_virt.o \

  amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

index 81fb3465e197..104bf600c85f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -885,6 +885,44 @@ static void amdgpu_vm_tlb_seq_cb(struct 
dma_fence *fence,

  kfree(tlb_cb);
  }
  +/**
+ * amdgpu_vm_tlb_flush - prepare TLB flush
+ *
+ * @params: parameters for update
+ * @fence: input fence to sync TLB flush with
+ * @tlb_cb: the callback structure
+ *
+ * Increments the tlb sequence to make sure that future CS execute a 
VM flush.

+ */
+static void
+amdgpu_vm_tlb_flush(struct amdgpu_vm_update_params *params,
+    struct dma_fence **fence,
+    struct amdgpu_vm_tlb_seq_struct *tlb_cb)
+{
+    struct amdgpu_vm *vm = params->vm;
+
+    if (!fence || !*fence)
+    return;
+
+    tlb_cb->vm = vm;
+    if (!dma_fence_add_callback(*fence, _cb->cb,
+    amdgpu_vm_tlb_seq_cb)) {
+    dma_fence_put(vm->last_tlb_flush);
+    vm->last_tlb_flush = dma_fence_get(*fence);
+    } else {
+    amdgpu_vm_tlb_seq_cb(NULL, _cb->cb);
+    }
+
+    /* Prepare a TLB flush fence to be attached to PTs */
+    if (!params->unlocked && vm->is_compute_context) {
+    amdgpu_vm_tlb_fence_create(params->adev, vm, fence);
+
+    /* Makes sure no PD/PT is freed before the flush */
+    dma_resv_add_fence(vm->root.bo->tbo.base.resv, *fence,
+   DMA_RESV_USAGE_BOOKKEEP);
+    }
+}
+
  /**
   * amdgpu_vm_update_range - update a range in the vm page table
   *
@@ -916,8 +954,8 @@ int amdgpu_vm_update_range(struct amdgpu_device 
*adev, struct amdgpu_vm *vm,

 struct ttm_resource *res, dma_addr_t *pages_addr,
 struct dma_fence **fence)
  {
-    struct amdgpu_vm_update_params params;
  struct amdgpu_vm_tlb_seq_struct *tlb_cb;
+    struct amdgpu_vm_update_params params;
  struct amdgpu_res_cursor cursor;
  enum amdgpu_sync_mode sync_mode;
 

Re: [PATCH v8] drm/amdgpu: sync page table freeing with tlb flush

[Sharma, Shashank]

On 18/03/2024 16:01, Christian König wrote:

Am 18.03.24 um 15:44 schrieb Shashank Sharma:

The idea behind this patch is to delay the freeing of PT entry objects
until the TLB flush is done.

This patch:
- Adds a tlb_flush_waitlist in amdgpu_vm_update_params which will 
keep the

   objects that need to be freed after tlb_flush.
- Adds PT entries in this list in amdgpu_vm_ptes_update after finding
   the PT entry.
- Changes functionality of amdgpu_vm_pt_free_dfs from (df_search + free)
   to simply freeing of the BOs, also renames it to
   amdgpu_vm_pt_free_list to reflect this same.
- Exports function amdgpu_vm_pt_free_list to be called directly.
- Calls amdgpu_vm_pt_free_list directly from amdgpu_vm_update_range.

V2: rebase
V4: Addressed review comments from Christian
 - add only locked PTEs entries in TLB flush waitlist.
 - do not create a separate function for list flush.
 - do not create a new lock for TLB flush.
 - there is no need to wait on tlb_flush_fence exclusively.

V5: Addressed review comments from Christian
 - change the amdgpu_vm_pt_free_dfs's functionality to simple 
freeing

   of the objects and rename it.
 - add all the PTE objects in params->tlb_flush_waitlist
 - let amdgpu_vm_pt_free_root handle the freeing of BOs 
independently

 - call amdgpu_vm_pt_free_list directly

V6: Rebase
V7: Rebase
V8: Added a NULL check to fix this backtrace issue:
[  415.351447] BUG: kernel NULL pointer dereference, address: 
0008

[  415.359245] #PF: supervisor write access in kernel mode
[  415.365081] #PF: error_code(0x0002) - not-present page
[  415.370817] PGD 101259067 P4D 101259067 PUD 10125a067 PMD 0
[  415.377140] Oops: 0002 [#1] PREEMPT SMP NOPTI
[  415.382004] CPU: 0 PID: 25481 Comm: test_with_MPI.e Tainted: 
G   OE 5.18.2-mi300-build-140423-ubuntu-22.04+ #24
[  415.394437] Hardware name: AMD Corporation Sh51p/Sh51p, BIOS 
RMO1001AS 02/21/2024

[  415.402797] RIP: 0010:amdgpu_vm_ptes_update+0x6fd/0xa10 [amdgpu]
[  415.409648] Code: 4c 89 ff 4d 8d 66 30 e8 f1 ed ff ff 48 85 db 74 
42 48 39 5d a0 74 40 48 8b 53 20 48 8b 4b 18 48 8d 43 18 48 8d 75 b0 
4c 89 ff <48

89 51 08 48 89 0a 49 8b 56 30 48 89 42 08 48 89 53 18 4c 89 63

[  415.430621] RSP: 0018:c9000401f990 EFLAGS: 00010287
[  415.436456] RAX: 888147bb82f0 RBX: 888147bb82d8 RCX: 

[  415.26] RDX:  RSI: c9000401fa30 RDI: 
888161f8
[  415.452397] RBP: c9000401fa80 R08:  R09: 
c9000401fa00
[  415.460368] R10: 0007f0cc R11: 0007f0c85000 R12: 
c9000401fb20
[  415.468340] R13: 0007f0d0 R14: c9000401faf0 R15: 
888161f8
[  415.476312] FS:  7f132ff89840() GS:889f87c0() 
knlGS:

[  415.485350] CS:  0010 DS:  ES:  CR0: 80050033
[  415.491767] CR2: 0008 CR3: 000161d46003 CR4: 
00770ef0

[  415.499738] PKRU: 5554
[  415.502750] Call Trace:
[  415.505482]  
[  415.507825]  amdgpu_vm_update_range+0x32a/0x880 [amdgpu]
[  415.513869]  amdgpu_vm_clear_freed+0x117/0x250 [amdgpu]
[  415.519814] amdgpu_amdkfd_gpuvm_unmap_memory_from_gpu+0x18c/0x250 
[amdgpu]

[  415.527729]  kfd_ioctl_unmap_memory_from_gpu+0xed/0x340 [amdgpu]
[  415.534551]  kfd_ioctl+0x3b6/0x510 [amdgpu]

Cc: Christian König 
Cc: Alex Deucher 
Cc: Felix Kuehling 
Cc: Rajneesh Bhardwaj 
Acked-by: Felix Kuehling 
Acked-by: Rajneesh Bhardwaj 
Tested-by: Rajneesh Bhardwaj 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    |  5 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |  7 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c | 58 +--
  3 files changed, 45 insertions(+), 25 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

index 26f1c3359642..eaa402f99fe0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -977,6 +977,7 @@ int amdgpu_vm_update_range(struct amdgpu_device 
*adev, struct amdgpu_vm *vm,

  params.unlocked = unlocked;
  params.needs_flush = flush_tlb;
  params.allow_override = allow_override;
+    INIT_LIST_HEAD(_flush_waitlist);
    /* Implicitly sync to command submissions in the same VM before
   * unmapping. Sync to moving fences before mapping.
@@ -1062,8 +1063,10 @@ int amdgpu_vm_update_range(struct 
amdgpu_device *adev, struct amdgpu_vm *vm,

  if (r)
  goto error_unlock;
  -    if (params.needs_flush)
+    if (params.needs_flush) {
  r = amdgpu_vm_tlb_flush(, fence);
+    amdgpu_vm_pt_free_list(adev, );


This is completed independent of the VM flush and should always be 
called.



+    }
    error_unlock:
  amdgpu_vm_eviction_unlock(vm);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h

index b0a4fe683352..54d7da396de0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h

Re: [PATCH v7 1/2] drm/amdgpu: implement TLB flush fence

[Sharma, Shashank]

On 18/03/2024 15:58, Christian König wrote:

Am 18.03.24 um 13:08 schrieb Shashank Sharma:

From: Christian Koenig 

The problem is that when (for example) 4k pages are replaced
with a single 2M page we need to wait for change to be flushed
out by invalidating the TLB before the PT can be freed.

Solve this by moving the TLB flush into a DMA-fence object which
can be used to delay the freeing of the PT BOs until it is signaled.

V2: (Shashank)
 - rebase
 - set dma_fence_error only in case of error
 - add tlb_flush fence only when PT/PD BO is locked (Felix)
 - use vm->pasid when f is NULL (Mukul)

V4: - add a wait for (f->dependency) in tlb_fence_work (Christian)
 - move the misplaced fence_create call to the end (Philip)

V5: - free the f->dependency properly

V6: (Shashank)
 - light code movement, moved all the clean-up in previous patch
 - introduce params.needs_flush and its usage in this patch
 - rebase without TLB HW sequence patch

V7:
    - Keep the vm->last_update_fence and tlb_cb code until
  we can fix the HW sequencing (Christian)
    - Move all the tlb_fence related code in a separate function so that
  its easier to read and review

Cc: Christian Koenig 
Cc: Felix Kuehling 
Cc: Rajneesh Bhardwaj 
Cc: Alex Deucher 
Acked-by: Felix Kuehling 
Acked-by: Rajneesh Bhardwaj 
Tested-by: Rajneesh Bhardwaj 
Reviewed-by: Shashank Sharma 
Signed-off-by: Christian Koenig 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |   3 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    |  68 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |   8 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_cpu.c    |   4 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_sdma.c   |   4 +
  .../gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c  | 112 ++
  7 files changed, 171 insertions(+), 30 deletions(-)
  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index 4536c8ad0e11..f24f11ac3e92 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -70,7 +70,8 @@ amdgpu-y += amdgpu_device.o amdgpu_doorbell_mgr.o 
amdgpu_kms.o \

  amdgpu_cs.o amdgpu_bios.o amdgpu_benchmark.o \
  atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
  atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
-    amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o 
amdgpu_pll.o \

+    amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_vm_tlb_fence.o \
+    amdgpu_ib.o amdgpu_pll.o \
  amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
  amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o 
amdgpu_virt.o \

  amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

index 81fb3465e197..26f1c3359642 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -885,6 +885,40 @@ static void amdgpu_vm_tlb_seq_cb(struct 
dma_fence *fence,

  kfree(tlb_cb);
  }
  +static int
+amdgpu_vm_tlb_flush(struct amdgpu_vm_update_params *params, struct 
dma_fence **fence)

+{
+    struct amdgpu_vm_tlb_seq_struct *tlb_cb;
+    struct amdgpu_vm *vm = params->vm;
+
+    if (!fence || !*fence)
+    return 0;
+
+    tlb_cb = kmalloc(sizeof(*tlb_cb), GFP_KERNEL);
+    if (!tlb_cb)
+    return -ENOMEM;


That won't work like this. The page tables are already updated, so you 
need to have the callback no matter what.


That's why the code previously allocated the tlb_cb structure before 
updating the page tables.



I could see from the previous code that there was also handling of cb 
being OOM, but I think your point is 'do not start the PT update until 
we have memory for callback', so I will accommodate that.


- Shashank



Regards,
Christian.


+
+    tlb_cb->vm = vm;
+    if (!dma_fence_add_callback(*fence, _cb->cb,
+    amdgpu_vm_tlb_seq_cb)) {
+    dma_fence_put(vm->last_tlb_flush);
+    vm->last_tlb_flush = dma_fence_get(*fence);
+    } else {
+    amdgpu_vm_tlb_seq_cb(NULL, _cb->cb);
+    }
+
+    /* Prepare a TLB flush fence to be attached to PTs */
+    if (!params->unlocked && vm->is_compute_context) {
+    amdgpu_vm_tlb_fence_create(params->adev, vm, fence);
+
+    /* Makes sure no PD/PT is freed before the flush */
+    dma_resv_add_fence(vm->root.bo->tbo.base.resv, *fence,
+   DMA_RESV_USAGE_BOOKKEEP);
+    }
+
+    return 0;
+}
+
  /**
   * amdgpu_vm_update_range - update a range in the vm page table
   *
@@ -917,7 +951,6 @@ int amdgpu_vm_update_range(struct amdgpu_device 
*adev, struct amdgpu_vm *vm,

 struct dma_fence **fence)
  {
  struct amdgpu_vm_update_params params;
-    struct amdgpu_vm_tlb_seq_struct *tlb_cb;
  struct 

Re: [PATCH v7 2/2] drm/amdgpu: sync page table freeing with tlb flush

[Sharma, Shashank]

[AMD Official Use Only - General]

Already sent a NULL check patch based on this backtrace, I am waiting for 
Rajneesh's feedback.

Regards
Shashank

From: Bhardwaj, Rajneesh 
Sent: Monday, March 18, 2024 3:04 PM
To: Sharma, Shashank ; amd-gfx@lists.freedesktop.org 

Cc: Koenig, Christian ; Deucher, Alexander 
; Kuehling, Felix 
Subject: Re: [PATCH v7 2/2] drm/amdgpu: sync page table freeing with tlb flush


HI Shashank

We'll probably need a v8 with the null pointer crash fixed i.e. before freeing 
the PT entries check for a valid entry before calling amdgpu_vm_pt_free. The 
crash is seen with device memory allocators but the system memory allocators 
are looking fine.



[  127.255863] [drm] Using MTYPE_RW for local memory
[  333.606136] hugetlbfs: test_with_MPI.e (25268): Using mlock ulimits for 
SHM_HUGETLB is obsolete
[  415.351447] BUG: kernel NULL pointer dereference, address: 0008
[  415.359245] #PF: supervisor write access in kernel mode
[  415.365081] #PF: error_code(0x0002) - not-present page
[  415.370817] PGD 101259067 P4D 101259067 PUD 10125a067 PMD 0
[  415.377140] Oops: 0002 [#1] PREEMPT SMP NOPTI
[  415.382004] CPU: 0 PID: 25481 Comm: test_with_MPI.e Tainted: G   OE  
   5.18.2-mi300-build-140423-ubuntu-22.04+ #24
[  415.394437] Hardware name: AMD Corporation Sh51p/Sh51p, BIOS RMO1001AS 
02/21/2024
[  415.402797] RIP: 0010:amdgpu_vm_ptes_update+0x6fd/0xa10 [amdgpu]
[  415.409648] Code: 4c 89 ff 4d 8d 66 30 e8 f1 ed ff ff 48 85 db 74 42 48 39 
5d a0 74 40 48 8b 53 20 48 8b 4b 18 48 8d 43 18 48 8d 75 b0 4c 89 ff <48
> 89 51 08 48 89 0a 49 8b 56 30 48 89 42 08 48 89 53 18 4c 89 63
[  415.430621] RSP: 0018:c9000401f990 EFLAGS: 00010287
[  415.436456] RAX: 888147bb82f0 RBX: 888147bb82d8 RCX: 
[  415.26] RDX:  RSI: c9000401fa30 RDI: 888161f8
[  415.452397] RBP: c9000401fa80 R08:  R09: c9000401fa00
[  415.460368] R10: 0007f0cc R11: 0007f0c85000 R12: c9000401fb20
[  415.468340] R13: 0007f0d0 R14: c9000401faf0 R15: 888161f8
[  415.476312] FS:  7f132ff89840() GS:889f87c0() 
knlGS:
[  415.485350] CS:  0010 DS:  ES:  CR0: 80050033
[  415.491767] CR2: 0008 CR3: 000161d46003 CR4: 00770ef0
[  415.499738] PKRU: 5554
[  415.502750] Call Trace:
[  415.505482]  
[  415.507825]  amdgpu_vm_update_range+0x32a/0x880 [amdgpu]
[  415.513869]  amdgpu_vm_clear_freed+0x117/0x250 [amdgpu]
[  415.519814]  amdgpu_amdkfd_gpuvm_unmap_memory_from_gpu+0x18c/0x250 [amdgpu]
[  415.527729]  kfd_ioctl_unmap_memory_from_gpu+0xed/0x340 [amdgpu]
[  415.534551]  kfd_ioctl+0x3b6/0x510 [amdgpu]
[  415.539324]  ? kfd_ioctl_get_dmabuf_info+0x1d0/0x1d0 [amdgpu]
[  415.545844]  ? __fget_light+0xc5/0x100
[  415.550037]  __x64_sys_ioctl+0x91/0xc0
[  415.554227]  do_syscall_64+0x5c/0x80
[  415.558223]  ? debug_smp_processor_id+0x17/0x20
[  415.563285]  ? fpregs_assert_state_consistent+0x23/0x60
[  415.569124]  ? exit_to_user_mode_prepare+0x45/0x190
[  415.574572]  ? ksys_write+0xce/0xe0




On 3/18/2024 8:08 AM, Shashank Sharma wrote:

The idea behind this patch is to delay the freeing of PT entry objects
until the TLB flush is done.

This patch:
- Adds a tlb_flush_waitlist in amdgpu_vm_update_params which will keep the
  objects that need to be freed after tlb_flush.
- Adds PT entries in this list in amdgpu_vm_ptes_update after finding
  the PT entry.
- Changes functionality of amdgpu_vm_pt_free_dfs from (df_search + free)
  to simply freeing of the BOs, also renames it to
  amdgpu_vm_pt_free_list to reflect this same.
- Exports function amdgpu_vm_pt_free_list to be called directly.
- Calls amdgpu_vm_pt_free_list directly from amdgpu_vm_update_range.

V2: rebase
V4: Addressed review comments from Christian
- add only locked PTEs entries in TLB flush waitlist.
- do not create a separate function for list flush.
- do not create a new lock for TLB flush.
- there is no need to wait on tlb_flush_fence exclusively.

V5: Addressed review comments from Christian
- change the amdgpu_vm_pt_free_dfs's functionality to simple freeing
  of the objects and rename it.
- add all the PTE objects in params->tlb_flush_waitlist
- let amdgpu_vm_pt_free_root handle the freeing of BOs independently
- call amdgpu_vm_pt_free_list directly

V6: Rebase
V7: Rebase

Cc: Christian König <mailto:christian.koe...@amd.com>
Cc: Alex Deucher <mailto:alexander.deuc...@amd.com>
Cc: Felix Kuehling <mailto:felix.kuehl...@amd.com>
Cc: Rajneesh Bhardwaj 
<mailto:rajneesh.bhard...@amd.com>
Acked-by: Felix Kuehling <mailto:felix.kuehl...@amd.com>
Acked-by: Rajneesh Bhardwaj 
<mailto:rajneesh.bhard...@amd.com>
Tested-by: Rajneesh Bhardwaj 
<mailto:rajneesh.bhard...@amd.com>
Signed-off-by: Shashank Sharma 
<mailto:shashank.sha...@amd.co

Re: [PATCH 2/2] drm:amdgpu: add firmware information of all IP's

[Sharma, Shashank]

On 14/03/2024 06:58, Khatri, Sunil wrote:


On 3/14/2024 2:06 AM, Alex Deucher wrote:
On Tue, Mar 12, 2024 at 8:42 AM Sunil Khatri  
wrote:

Add firmware version information of each
IP and each instance where applicable.


Is there a way we can share some common code with devcoredump,
debugfs, and the info IOCTL?  All three places need to query this
information and the same logic is repeated in each case.


Hello Alex,

Yes you re absolutely right the same information is being retrieved 
again as done in debugfs. I can reorganize the code so same function 
could be used by debugfs and devcoredump but this is exactly what i 
tried to avoid here. I did try to use minimum functionality in 
devcoredump without shuffling a lot of code here and there.


Also our devcoredump is implemented in amdgpu_reset.c and not all the 
information is available here and there we might have to include lot 
of header and cross functions in amdgpu_reset until we want a 
dedicated file for devcoredump.



I think Alex is suggesting to have one common backend to generate all 
the core debug info, and then different wrapper functions which can pack 
this raw info into the packets aligned with respective infra like 
devcore/debugfs/info IOCTL, which seems like a good idea to me.


If you think you need a new file for this backend it should be fine.


something like:

amdgpu_debug_core.c::

struct amdgpu_core_debug_info {

/* Superset of all the info you are collecting from HW */

};

- amdgpu_debug_generate_core_info

{

/* This function collects the core debug info from HW and saves in 
amdgpu_core_debug_info,


  we can update this periodically regardless of a request */

}

and then:

devcore_info *amdgpu_debug_pack_for_devcore(core_debug_info)

{

/* convert core debug info into devcore aligned format/data */

}

ioctl_info *amdgpu_debug_pack_for_info_ioctl(core_debug_info)

{

/* convert core debug info into info IOCTL aligned format/data */

}

debugfs_info *amdgpu_debug_pack_for_debugfs(core_debug_info)

{

/* convert core debug info into debugfs aligned format/data */

}

- Shashank





Info IOCTL does have a lot of information which also is in pipeline to 
be dumped but this if we want to reuse the functionality of IOCTL we 
need to reorganize a lot of code.


If that is the need of the hour i could work on that. Please let me know.

This is my suggestion if it makes sense:

1. If we want to reuse a lot of functionality then we need to 
modularize some of the functions further so they could be consumed 
directly by devcoredump.
2. We should also have a dedicated file for devcoredump.c/.h so its 
easy to include headers of needed functionality cleanly and easy to 
expand devcoredump.
3. based on the priority and importance of this task we can add 
information else some repetition is a real possibility.




Alex



Signed-off-by: Sunil Khatri 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c | 122 
++

  1 file changed, 122 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c

index 611fdb90a1fc..78ddc58aef67 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c
@@ -168,6 +168,123 @@ void amdgpu_coredump(struct amdgpu_device 
*adev, bool vram_lost,

  {
  }
  #else
+static void amdgpu_devcoredump_fw_info(struct amdgpu_device *adev, 
struct drm_printer *p)

+{
+   uint32_t version;
+   uint32_t feature;
+   uint8_t smu_program, smu_major, smu_minor, smu_debug;
+
+   drm_printf(p, "VCE feature version: %u, fw version: 0x%08x\n",
+  adev->vce.fb_version, adev->vce.fw_version);
+   drm_printf(p, "UVD feature version: %u, fw version: 0x%08x\n",
+  0, adev->uvd.fw_version);
+   drm_printf(p, "GMC feature version: %u, fw version: 0x%08x\n",
+  0, adev->gmc.fw_version);
+   drm_printf(p, "ME feature version: %u, fw version: 0x%08x\n",
+  adev->gfx.me_feature_version, 
adev->gfx.me_fw_version);

+   drm_printf(p, "PFP feature version: %u, fw version: 0x%08x\n",
+  adev->gfx.pfp_feature_version, 
adev->gfx.pfp_fw_version);

+   drm_printf(p, "CE feature version: %u, fw version: 0x%08x\n",
+  adev->gfx.ce_feature_version, 
adev->gfx.ce_fw_version);

+   drm_printf(p, "RLC feature version: %u, fw version: 0x%08x\n",
+  adev->gfx.rlc_feature_version, 
adev->gfx.rlc_fw_version);

+
+   drm_printf(p, "RLC SRLC feature version: %u, fw version: 
0x%08x\n",

+  adev->gfx.rlc_srlc_feature_version,
+  adev->gfx.rlc_srlc_fw_version);
+   drm_printf(p, "RLC SRLG feature version: %u, fw version: 
0x%08x\n",

+  adev->gfx.rlc_srlg_feature_version,
+  adev->gfx.rlc_srlg_fw_version);
+   drm_printf(p, "RLC SRLS feature version: %u, fw version: 
0x%08x\n",

+  

Re: [PATCH v5 1/2] drm/amdgpu: implement TLB flush fence

[Sharma, Shashank]

On 12/03/2024 09:31, Christian König wrote:

Am 11.03.24 um 15:37 schrieb Sharma, Shashank:


On 07/03/2024 20:22, Philip Yang wrote:



On 2024-03-06 09:41, Shashank Sharma wrote:

From: Christian König

The problem is that when (for example) 4k pages are replaced
with a single 2M page we need to wait for change to be flushed
out by invalidating the TLB before the PT can be freed.

Solve this by moving the TLB flush into a DMA-fence object which
can be used to delay the freeing of the PT BOs until it is signaled.

V2: (Shashank)
 - rebase
 - set dma_fence_error only in case of error
 - add tlb_flush fence only when PT/PD BO is locked (Felix)
 - use vm->pasid when f is NULL (Mukul)

V4: - add a wait for (f->dependency) in tlb_fence_work (Christian)
 - move the misplaced fence_create call to the end (Philip)

V5: - free the f->dependency properly (Christian)

Cc: Christian Koenig
Cc: Felix Kuehling
Cc: Rajneesh Bhardwaj
Cc: Alex Deucher
Reviewed-by: Shashank Sharma
Signed-off-by: Christian König
Signed-off-by: Shashank Sharma
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |   3 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    |  10 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |   4 +
  .../gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c  | 112 
++

  4 files changed, 128 insertions(+), 1 deletion(-)
  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index fa26a4e3a99d..91ab4cf29b5b 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -70,7 +70,8 @@ amdgpu-y += amdgpu_device.o amdgpu_doorbell_mgr.o 
amdgpu_kms.o \

  amdgpu_cs.o amdgpu_bios.o amdgpu_benchmark.o \
  atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
  atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
-    amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o 
amdgpu_pll.o \
+    amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o 
amdgpu_vm_tlb_fence.o \

+    amdgpu_ib.o amdgpu_pll.o \
  amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
  amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o 
amdgpu_virt.o \

  amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

index 0960e0a665d3..310aae6fb49b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -988,6 +988,15 @@ int amdgpu_vm_update_range(struct 
amdgpu_device *adev, struct amdgpu_vm *vm,

    r = vm->update_funcs->commit(, fence);
  +    /* Prepare a TLB flush fence to be attached to PTs */
+    if (!unlocked && params.needs_flush && vm->is_compute_context) {
+    amdgpu_vm_tlb_fence_create(adev, vm, fence);
+
+    /* Makes sure no PD/PT is freed before the flush */
+ dma_resv_add_fence(vm->root.bo->tbo.base.resv, *fence,
+   DMA_RESV_USAGE_BOOKKEEP);
+    }
+
  error_unlock:
  amdgpu_vm_eviction_unlock(vm);
  drm_dev_exit(idx);
@@ -2237,6 +2246,7 @@ int amdgpu_vm_init(struct amdgpu_device 
*adev, struct amdgpu_vm *vm,

    mutex_init(>eviction_lock);
  vm->evicting = false;
+    vm->tlb_fence_context = dma_fence_context_alloc(1);
    r = amdgpu_vm_pt_create(adev, vm, adev->vm_manager.root_level,
  false, , xcp_id);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h

index 64b3f69efa57..298f604b8e5f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -341,6 +341,7 @@ struct amdgpu_vm {
  atomic64_t    tlb_seq;
  uint64_t    tlb_seq_va;
  uint64_t    *tlb_seq_cpu_addr;
+    uint64_t    tlb_fence_context;
    atomic64_t    kfd_last_flushed_seq;
  @@ -594,5 +595,8 @@ void amdgpu_vm_update_fault_cache(struct 
amdgpu_device *adev,

    uint64_t addr,
    uint32_t status,
    unsigned int vmhub);
+void amdgpu_vm_tlb_fence_create(struct amdgpu_device *adev,
+ struct amdgpu_vm *vm,
+ struct dma_fence **fence);
    #endif
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c

new file mode 100644
index ..51cddfa3f1e8
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c
@@ -0,0 +1,112 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2023 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person 
obtaining a
+ * copy of this software and associated documentation files (the 
"Software"),
+ * to deal in the Software without restriction, including without 
limitation
+ * the rights to use, copy, modify, merge, publish, distribute, 
sublicense,
+ * and/or sell copies of

Re: [PATCH v5 1/2] drm/amdgpu: implement TLB flush fence

[Sharma, Shashank]

On 07/03/2024 20:22, Philip Yang wrote:



On 2024-03-06 09:41, Shashank Sharma wrote:

From: Christian König

The problem is that when (for example) 4k pages are replaced
with a single 2M page we need to wait for change to be flushed
out by invalidating the TLB before the PT can be freed.

Solve this by moving the TLB flush into a DMA-fence object which
can be used to delay the freeing of the PT BOs until it is signaled.

V2: (Shashank)
 - rebase
 - set dma_fence_error only in case of error
 - add tlb_flush fence only when PT/PD BO is locked (Felix)
 - use vm->pasid when f is NULL (Mukul)

V4: - add a wait for (f->dependency) in tlb_fence_work (Christian)
 - move the misplaced fence_create call to the end (Philip)

V5: - free the f->dependency properly (Christian)

Cc: Christian Koenig
Cc: Felix Kuehling
Cc: Rajneesh Bhardwaj
Cc: Alex Deucher
Reviewed-by: Shashank Sharma
Signed-off-by: Christian König
Signed-off-by: Shashank Sharma
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |   3 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c|  10 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h|   4 +
  .../gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c  | 112 ++
  4 files changed, 128 insertions(+), 1 deletion(-)
  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile
index fa26a4e3a99d..91ab4cf29b5b 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -70,7 +70,8 @@ amdgpu-y += amdgpu_device.o amdgpu_doorbell_mgr.o 
amdgpu_kms.o \
amdgpu_cs.o amdgpu_bios.o amdgpu_benchmark.o \
atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
-   amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o amdgpu_pll.o \
+   amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_vm_tlb_fence.o \
+   amdgpu_ib.o amdgpu_pll.o \
amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o amdgpu_virt.o \
amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 0960e0a665d3..310aae6fb49b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -988,6 +988,15 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, 
struct amdgpu_vm *vm,
  
  	r = vm->update_funcs->commit(, fence);
  
+	/* Prepare a TLB flush fence to be attached to PTs */

+   if (!unlocked && params.needs_flush && vm->is_compute_context) {
+   amdgpu_vm_tlb_fence_create(adev, vm, fence);
+
+   /* Makes sure no PD/PT is freed before the flush */
+   dma_resv_add_fence(vm->root.bo->tbo.base.resv, *fence,
+  DMA_RESV_USAGE_BOOKKEEP);
+   }
+
  error_unlock:
amdgpu_vm_eviction_unlock(vm);
drm_dev_exit(idx);
@@ -2237,6 +2246,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct 
amdgpu_vm *vm,
  
  	mutex_init(>eviction_lock);

vm->evicting = false;
+   vm->tlb_fence_context = dma_fence_context_alloc(1);
  
  	r = amdgpu_vm_pt_create(adev, vm, adev->vm_manager.root_level,

false, , xcp_id);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index 64b3f69efa57..298f604b8e5f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -341,6 +341,7 @@ struct amdgpu_vm {
atomic64_t  tlb_seq;
uint64_ttlb_seq_va;
uint64_t*tlb_seq_cpu_addr;
+   uint64_ttlb_fence_context;
  
  	atomic64_t		kfd_last_flushed_seq;
  
@@ -594,5 +595,8 @@ void amdgpu_vm_update_fault_cache(struct amdgpu_device *adev,

  uint64_t addr,
  uint32_t status,
  unsigned int vmhub);
+void amdgpu_vm_tlb_fence_create(struct amdgpu_device *adev,
+struct amdgpu_vm *vm,
+struct dma_fence **fence);
  
  #endif

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c
new file mode 100644
index ..51cddfa3f1e8
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c
@@ -0,0 +1,112 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2023 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * 

Re: [PATCH v5 1/2] drm/amdgpu: implement TLB flush fence

[Sharma, Shashank]

On 07/03/2024 00:54, Felix Kuehling wrote:


On 2024-03-06 09:41, Shashank Sharma wrote:

From: Christian König 

The problem is that when (for example) 4k pages are replaced
with a single 2M page we need to wait for change to be flushed
out by invalidating the TLB before the PT can be freed.

Solve this by moving the TLB flush into a DMA-fence object which
can be used to delay the freeing of the PT BOs until it is signaled.

V2: (Shashank)
 - rebase
 - set dma_fence_error only in case of error
 - add tlb_flush fence only when PT/PD BO is locked (Felix)
 - use vm->pasid when f is NULL (Mukul)

V4: - add a wait for (f->dependency) in tlb_fence_work (Christian)
 - move the misplaced fence_create call to the end (Philip)

V5: - free the f->dependency properly (Christian)

Cc: Christian Koenig 
Cc: Felix Kuehling 
Cc: Rajneesh Bhardwaj 
Cc: Alex Deucher 
Reviewed-by: Shashank Sharma 
Signed-off-by: Christian König 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |   3 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    |  10 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |   4 +
  .../gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c  | 112 ++
  4 files changed, 128 insertions(+), 1 deletion(-)
  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index fa26a4e3a99d..91ab4cf29b5b 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -70,7 +70,8 @@ amdgpu-y += amdgpu_device.o amdgpu_doorbell_mgr.o 
amdgpu_kms.o \

  amdgpu_cs.o amdgpu_bios.o amdgpu_benchmark.o \
  atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
  atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
-    amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o 
amdgpu_pll.o \

+    amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_vm_tlb_fence.o \
+    amdgpu_ib.o amdgpu_pll.o \
  amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
  amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o 
amdgpu_virt.o \

  amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

index 0960e0a665d3..310aae6fb49b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -988,6 +988,15 @@ int amdgpu_vm_update_range(struct amdgpu_device 
*adev, struct amdgpu_vm *vm,

    r = vm->update_funcs->commit(, fence);
  +    /* Prepare a TLB flush fence to be attached to PTs */
+    if (!unlocked && params.needs_flush && vm->is_compute_context) {
+    amdgpu_vm_tlb_fence_create(adev, vm, fence);


This schedules a TLB flush after "fence" signals and replaces "fence" 
with a new one that will signal after the TLB flush is done. That part 
I understand.


I'm not sure why this only applies to compute contexts.



+
+    /* Makes sure no PD/PT is freed before the flush */
+    dma_resv_add_fence(vm->root.bo->tbo.base.resv, *fence,
+   DMA_RESV_USAGE_BOOKKEEP);


But what's the point of adding the fence to the page table 
reservation? This is after the BOs have already been freed. Maybe it 
would make more sense to move this into the next patch, where the 
freeing is done after this point.


To make it easier for code review, the split of the patches is like:
- one patch introduces function creating tlb_flush_fence and uses it

- the second patch does the rework and movement of freeing of the buffer 
after the patch attach.


If we move this change into next patch, in this patch we will just 
create the fence, where one can argue why create the fence if no one is 
using it.


May be, we can make 'changes in freeing of buffers' as first patch in 
sequence, and make this second patch in the series, so that you know the 
background of changes better.


- Shashank



Regards,
  Felix



+    }
+
  error_unlock:
  amdgpu_vm_eviction_unlock(vm);
  drm_dev_exit(idx);
@@ -2237,6 +2246,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, 
struct amdgpu_vm *vm,

    mutex_init(>eviction_lock);
  vm->evicting = false;
+    vm->tlb_fence_context = dma_fence_context_alloc(1);
    r = amdgpu_vm_pt_create(adev, vm, adev->vm_manager.root_level,
  false, , xcp_id);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h

index 64b3f69efa57..298f604b8e5f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -341,6 +341,7 @@ struct amdgpu_vm {
  atomic64_t    tlb_seq;
  uint64_t    tlb_seq_va;
  uint64_t    *tlb_seq_cpu_addr;
+    uint64_t    tlb_fence_context;
    atomic64_t    kfd_last_flushed_seq;
  @@ -594,5 +595,8 @@ void amdgpu_vm_update_fault_cache(struct 
amdgpu_device *adev,

    uint64_t addr,
   

Re: [PATCH v4 2/2] drm/amdgpu: sync page table freeing with tlb flush

[Sharma, Shashank]

On 01/03/2024 14:29, Christian König wrote:



Am 01.03.24 um 12:07 schrieb Shashank Sharma:

The idea behind this patch is to delay the freeing of PT entry objects
until the TLB flush is done.

This patch:
- Adds a tlb_flush_waitlist which will keep the objects that need to be
   freed after tlb_flush
- Adds PT entries in this list in amdgpu_vm_pt_free_dfs, instead of 
freeing

   them immediately.
- Exports function amdgpu_vm_pt_free to be called dircetly.
- Adds a 'force' input bool to amdgpu_vm_pt_free_dfs to differentiate
   between immediate freeing of the BOs (like from
   amdgpu_vm_pt_free_root) vs delayed freeing.

V2: rebase
V4: (Christian)
 - add only locked PTEs entries in TLB flush waitlist.
 - do not create a separate function for list flush.
 - do not create a new lock for TLB flush.
 - there is no need to wait on tlb_flush_fence exclusively.

Cc: Christian König 
Cc: Alex Deucher 
Cc: Felix Kuehling 
Cc: Rajneesh Bhardwaj 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    | 10 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |  4 
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c | 21 ++---
  3 files changed, 28 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

index 310aae6fb49b..94581a1fe34f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -990,11 +990,20 @@ int amdgpu_vm_update_range(struct amdgpu_device 
*adev, struct amdgpu_vm *vm,

    /* Prepare a TLB flush fence to be attached to PTs */
  if (!unlocked && params.needs_flush && vm->is_compute_context) {
+    struct amdgpu_vm_bo_base *entry, *next;
+
  amdgpu_vm_tlb_fence_create(adev, vm, fence);
    /* Makes sure no PD/PT is freed before the flush */
  dma_resv_add_fence(vm->root.bo->tbo.base.resv, *fence,
 DMA_RESV_USAGE_BOOKKEEP);
+
+    if (list_empty(>tlb_flush_waitlist))
+    goto error_unlock;
+
+    /* Now actually free the waitlist */
+    list_for_each_entry_safe(entry, next, 
>tlb_flush_waitlist, vm_status)

+    amdgpu_vm_pt_free(entry);


This needs to be outside of the "if". We also need to free the PDs/PTs 
in non compute VMs.



Noted,




  }
    error_unlock:
@@ -2214,6 +2223,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, 
struct amdgpu_vm *vm,

  INIT_LIST_HEAD(>pt_freed);
  INIT_WORK(>pt_free_work, amdgpu_vm_pt_free_work);
  INIT_KFIFO(vm->faults);
+    INIT_LIST_HEAD(>tlb_flush_waitlist);
    r = amdgpu_seq64_alloc(adev, >tlb_seq_va, 
>tlb_seq_cpu_addr);

  if (r)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h

index 298f604b8e5f..ba374c2c61bd 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -343,6 +343,9 @@ struct amdgpu_vm {
  uint64_t    *tlb_seq_cpu_addr;
  uint64_t    tlb_fence_context;
  +    /* temporary storage of PT BOs until the TLB flush */
+    struct list_head    tlb_flush_waitlist;
+
  atomic64_t    kfd_last_flushed_seq;
    /* How many times we had to re-generate the page tables */
@@ -545,6 +548,7 @@ int amdgpu_vm_ptes_update(struct 
amdgpu_vm_update_params *params,

    uint64_t start, uint64_t end,
    uint64_t dst, uint64_t flags);
  void amdgpu_vm_pt_free_work(struct work_struct *work);
+void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry);
    #if defined(CONFIG_DEBUG_FS)
  void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct 
seq_file *m);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c

index 95dc0afdaffb..cb14e5686c0f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
@@ -636,7 +636,7 @@ static int amdgpu_vm_pt_alloc(struct 
amdgpu_device *adev,

   *
   * @entry: PDE to free
   */
-static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry)
+void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry)
  {
  struct amdgpu_bo *shadow;
  @@ -685,13 +685,15 @@ void amdgpu_vm_pt_free_work(struct 
work_struct *work)

   * @vm: amdgpu vm structure
   * @start: optional cursor where to start freeing PDs/PTs
   * @unlocked: vm resv unlock status
+ * @force: force free all PDs/PTs without waiting for TLB flush
   *
   * Free the page directory or page table level and all sub levels.
   */
  static void amdgpu_vm_pt_free_dfs(struct amdgpu_device *adev,
    struct amdgpu_vm *vm,
    struct amdgpu_vm_pt_cursor *start,
-  bool unlocked)
+  bool unlocked,
+  bool force)
  {
  struct amdgpu_vm_pt_cursor cursor;
  struct amdgpu_vm_bo_base *entry;
@@ -708,11 +710,15 @@ static void amdgpu_vm_pt_free_dfs(struct 
amdgpu_device *adev,

  return;
  

Re: [PATCH v3] drm/amdgpu: change vm->task_info handling

[Sharma, Shashank]

On 01/03/2024 18:07, Felix Kuehling wrote:

On 2024-02-05 12:05, Shashank Sharma wrote:

This patch changes the handling and lifecycle of vm->task_info object.
The major changes are:
- vm->task_info is a dynamically allocated ptr now, and its uasge is
   reference counted.
- introducing two new helper funcs for task_info lifecycle management
 - amdgpu_vm_get_task_info: reference counts up task_info before
   returning this info
 - amdgpu_vm_put_task_info: reference counts down task_info
- last put to task_info() frees task_info from the vm.

This patch also does logistical changes required for existing usage
of vm->task_info.

V2: Do not block all the prints when task_info not found (Felix)
V3: (Felix)
- Fix wrong indentation
- No debug message for -ENOMEM
- Add NULL check for task_info
- Do not duplicate the debug messages (ti vs no ti)
- Get first reference of task_info in vm_init(), put last
  in vm_fini()

Cc: Christian Koenig
Cc: Alex Deucher
Cc: Felix Kuehling
Signed-off-by: Shashank Sharma


One nit-pick and one bug inline. With those fixed, the patch

Reviewed-by: Felix Kuehling 



---
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |   9 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c |  18 ++-
  drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c   |  12 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 158 ++--
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  21 ++-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c   |   2 +-
  drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c  |  24 +--
  drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c  |  23 +--
  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c   |  20 ++-
  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   |  23 +--
  drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c  |  23 +--
  drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c|  22 +--
  drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c |  20 +--
  13 files changed, 251 insertions(+), 124 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index 0e61ebdb3f3e..f9eb12697b95 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
@@ -1775,9 +1775,14 @@ static int amdgpu_debugfs_vm_info_show(struct seq_file 
*m, void *unused)
list_for_each_entry(file, >filelist, lhead) {
struct amdgpu_fpriv *fpriv = file->driver_priv;
struct amdgpu_vm *vm = >vm;
+   struct amdgpu_task_info *ti;
+
+   ti = amdgpu_vm_get_task_info_vm(vm);
+   if (ti) {
+   seq_printf(m, "pid:%d\tProcess:%s --\n", ti->pid, 
ti->process_name);
+   amdgpu_vm_put_task_info(ti);
+   }
  
-		seq_printf(m, "pid:%d\tProcess:%s --\n",

-   vm->task_info.pid, vm->task_info.process_name);
r = amdgpu_bo_reserve(vm->root.bo, true);
if (r)
break;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
index 1f357198533f..e6e6d56398f2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
@@ -35,7 +35,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct 
drm_sched_job *s_job)
  {
struct amdgpu_ring *ring = to_amdgpu_ring(s_job->sched);
struct amdgpu_job *job = to_amdgpu_job(s_job);
-   struct amdgpu_task_info ti;
+   struct amdgpu_task_info *ti;
struct amdgpu_device *adev = ring->adev;
int idx;
int r;
@@ -48,7 +48,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct 
drm_sched_job *s_job)
return DRM_GPU_SCHED_STAT_ENODEV;
}
  
-	memset(, 0, sizeof(struct amdgpu_task_info));

+
adev->job_hang = true;
  
  	if (amdgpu_gpu_recovery &&

@@ -58,12 +58,16 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct 
drm_sched_job *s_job)
goto exit;
}
  
-	amdgpu_vm_get_task_info(ring->adev, job->pasid, );

DRM_ERROR("ring %s timeout, signaled seq=%u, emitted seq=%u\n",
- job->base.sched->name, atomic_read(>fence_drv.last_seq),
- ring->fence_drv.sync_seq);
-   DRM_ERROR("Process information: process %s pid %d thread %s pid %d\n",
- ti.process_name, ti.tgid, ti.task_name, ti.pid);
+  job->base.sched->name, 
atomic_read(>fence_drv.last_seq),
+  ring->fence_drv.sync_seq);
+
+   ti = amdgpu_vm_get_task_info_pasid(ring->adev, job->pasid);
+   if (ti) {
+   DRM_ERROR("Process information: process %s pid %d thread %s pid 
%d\n",
+ ti->process_name, ti->tgid, ti->task_name, ti->pid);
+   amdgpu_vm_put_task_info(ti);
+   }
  
  	dma_fence_set_error(_job->s_fence->finished, -ETIME);
  
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c 

Re: [PATCH v4 1/2] drm/amdgpu: implement TLB flush fence

[Sharma, Shashank]

On 01/03/2024 13:59, Christian König wrote:

Am 01.03.24 um 12:07 schrieb Shashank Sharma:

From: Christian König 

The problem is that when (for example) 4k pages are replaced
with a single 2M page we need to wait for change to be flushed
out by invalidating the TLB before the PT can be freed.

Solve this by moving the TLB flush into a DMA-fence object which
can be used to delay the freeing of the PT BOs until it is signaled.

V2: (Shashank)
 - rebase
 - set dma_fence_error only in case of error
 - add tlb_flush fence only when PT/PD BO is locked (Felix)
 - use vm->pasid when f is NULL (Mukul)

V4: - add a wait for (f->dependency) in tlb_fence_work (Christian)
 - move the misplaced fence_create call to the end (Philip)

Cc: Christian Koenig 
Cc: Felix Kuehling 
Cc: Rajneesh Bhardwaj 
Cc: Alex Deucher 
Signed-off-by: Christian König 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |   3 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    |  10 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |   4 +
  .../gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c  | 111 ++
  4 files changed, 127 insertions(+), 1 deletion(-)
  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index fa26a4e3a99d..91ab4cf29b5b 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -70,7 +70,8 @@ amdgpu-y += amdgpu_device.o amdgpu_doorbell_mgr.o 
amdgpu_kms.o \

  amdgpu_cs.o amdgpu_bios.o amdgpu_benchmark.o \
  atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
  atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
-    amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o 
amdgpu_pll.o \

+    amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_vm_tlb_fence.o \
+    amdgpu_ib.o amdgpu_pll.o \
  amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
  amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o 
amdgpu_virt.o \

  amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

index 0960e0a665d3..310aae6fb49b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -988,6 +988,15 @@ int amdgpu_vm_update_range(struct amdgpu_device 
*adev, struct amdgpu_vm *vm,

    r = vm->update_funcs->commit(, fence);
  +    /* Prepare a TLB flush fence to be attached to PTs */
+    if (!unlocked && params.needs_flush && vm->is_compute_context) {
+    amdgpu_vm_tlb_fence_create(adev, vm, fence);
+
+    /* Makes sure no PD/PT is freed before the flush */
+    dma_resv_add_fence(vm->root.bo->tbo.base.resv, *fence,
+   DMA_RESV_USAGE_BOOKKEEP);
+    }
+
  error_unlock:
  amdgpu_vm_eviction_unlock(vm);
  drm_dev_exit(idx);
@@ -2237,6 +2246,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, 
struct amdgpu_vm *vm,

    mutex_init(>eviction_lock);
  vm->evicting = false;
+    vm->tlb_fence_context = dma_fence_context_alloc(1);
    r = amdgpu_vm_pt_create(adev, vm, adev->vm_manager.root_level,
  false, , xcp_id);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h

index 64b3f69efa57..298f604b8e5f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -341,6 +341,7 @@ struct amdgpu_vm {
  atomic64_t    tlb_seq;
  uint64_t    tlb_seq_va;
  uint64_t    *tlb_seq_cpu_addr;
+    uint64_t    tlb_fence_context;
    atomic64_t    kfd_last_flushed_seq;
  @@ -594,5 +595,8 @@ void amdgpu_vm_update_fault_cache(struct 
amdgpu_device *adev,

    uint64_t addr,
    uint32_t status,
    unsigned int vmhub);
+void amdgpu_vm_tlb_fence_create(struct amdgpu_device *adev,
+ struct amdgpu_vm *vm,
+ struct dma_fence **fence);
    #endif
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c

new file mode 100644
index ..54c33c24fa46
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c
@@ -0,0 +1,111 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2023 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person 
obtaining a
+ * copy of this software and associated documentation files (the 
"Software"),
+ * to deal in the Software without restriction, including without 
limitation
+ * the rights to use, copy, modify, merge, publish, distribute, 
sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom 
the

+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be 
included in

+ * all 

Re: [PATCH v3 3/3] drm/amdgpu: sync page table freeing with tlb flush

[Sharma, Shashank]

On 26/02/2024 17:52, Philip Yang wrote:



On 2024-02-23 08:42, Shashank Sharma wrote:

This patch:
- adds a new list in amdgou_vm to hold the VM PT entries being freed
- waits for the TLB flush using the vm->tlb_flush_fence
- actually frees the PT BOs

V2: rebase
V3: Do not attach the tlb_fence to the entries, rather add the entries
 to a list and delay their freeing (Christian)

Cc: Christian König
Cc: Alex Deucher
Cc: Felix Kuehling
Cc: Rajneesh Bhardwaj
Signed-off-by: Shashank Sharma
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c|  6 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h|  6 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c | 51 ---
  3 files changed, 58 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 67c690044b97..eebb73f2c2ef 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -939,6 +939,10 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, 
struct amdgpu_vm *vm,
/* Makes sure no PD/PT is freed before the flush */
dma_resv_add_fence(vm->root.bo->tbo.base.resv, *fence,
   DMA_RESV_USAGE_BOOKKEEP);
+
+   mutex_lock(>tlb_fence_lock);
+   vm->tlb_fence_last = *fence;
+   mutex_unlock(>tlb_fence_lock);
}
  
  	amdgpu_res_first(pages_addr ? NULL : res, offset,

@@ -2212,6 +2216,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct 
amdgpu_vm *vm,
INIT_LIST_HEAD(>freed);
INIT_LIST_HEAD(>done);
INIT_LIST_HEAD(>pt_freed);
+   INIT_LIST_HEAD(>tlb_flush_waitlist);
INIT_WORK(>pt_free_work, amdgpu_vm_pt_free_work);
INIT_KFIFO(vm->faults);
  
@@ -2244,6 +2249,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,

vm->last_unlocked = dma_fence_get_stub();
vm->generation = 0;
  
+	mutex_init(>tlb_fence_lock);

mutex_init(>eviction_lock);
vm->evicting = false;
vm->tlb_fence_context = dma_fence_context_alloc(1);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index 8e6fd25d07b7..77f10ed80973 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -334,6 +334,10 @@ struct amdgpu_vm {
uint64_t*tlb_seq_cpu_addr;
uint64_ttlb_fence_context;
  
+	struct mutex 		tlb_fence_lock;

+   struct dma_fence*tlb_fence_last;
+   struct list_headtlb_flush_waitlist;
+
atomic64_t  kfd_last_flushed_seq;
  
  	/* How many times we had to re-generate the page tables */

@@ -379,6 +383,8 @@ struct amdgpu_vm {
  
  	/* cached fault info */

struct amdgpu_vm_fault_info fault_info;
+
+   int count_bos;
  };
  
  struct amdgpu_vm_manager {

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
index 95dc0afdaffb..57ea95c5c085 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
@@ -643,13 +643,13 @@ static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base 
*entry)
if (!entry->bo)
return;
  
-	entry->bo->vm_bo = NULL;

shadow = amdgpu_bo_shadowed(entry->bo);
if (shadow) {
ttm_bo_set_bulk_move(>tbo, NULL);
amdgpu_bo_unref();
}
ttm_bo_set_bulk_move(>bo->tbo, NULL);
+   entry->bo->vm_bo = NULL;
  
  	spin_lock(>vm->status_lock);

list_del(>vm_status);
@@ -657,6 +657,38 @@ static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base 
*entry)
amdgpu_bo_unref(>bo);
  }
  
+static void amdgpu_vm_pt_flush_waitlist(struct amdgpu_vm *vm)

+{
+   struct amdgpu_vm_bo_base *entry, *next;
+   LIST_HEAD(tlb_flush_waitlist);
+
+   if (!vm || list_empty(>tlb_flush_waitlist))
+   return;
+
+   /* Wait for pending TLB flush before freeing PT BOs */
+   mutex_lock(>tlb_fence_lock);
+   if (vm->tlb_fence_last && !dma_fence_is_signaled(vm->tlb_fence_last)) {
+   if (dma_fence_wait_timeout(vm->tlb_fence_last, false,
+  MAX_SCHEDULE_TIMEOUT) <= 0) {
+   DRM_ERROR("Timedout waiting for TLB flush, not freeing PT 
BOs\n");
+   mutex_unlock(>tlb_fence_lock);
+   return;
+   }
+
+   vm->tlb_fence_last = NULL;
+   }
+
+   /* Save the waitlist locally and reset the flushlist */
+   list_splice_init(>tlb_flush_waitlist, _flush_waitlist);
+   mutex_unlock(>tlb_fence_lock);
+
+   /* Now free the entries */
+   list_for_each_entry_safe(entry, next, _flush_waitlist, vm_status) {
+   if (entry)
+   amdgpu_vm_pt_free(entry);
+   }
+}
+
  void amdgpu_vm_pt_free_work(struct work_struct 

Re: [PATCH v3 1/3] drm/amdgpu: replace TLB seq callback with HW seq

[Sharma, Shashank]

[AMD Official Use Only - General]

Please feel free to use:
Reviewed-by: Shashank Sharma 

Regards
Shashank

From: Christian König 
Sent: Monday, February 26, 2024 3:45 PM
To: Sharma, Shashank ; amd-gfx@lists.freedesktop.org 

Cc: Koenig, Christian ; Deucher, Alexander 
; Kuehling, Felix ; 
Bhardwaj, Rajneesh 
Subject: Re: [PATCH v3 1/3] drm/amdgpu: replace TLB seq callback with HW seq

Am 23.02.24 um 14:42 schrieb Shashank Sharma:
> From: Christian König 
>
> The callback we installed for the SDMA update were actually pretty
> horrible. since we now have seq64 use that one and HW seq writes
> instead.
>
> V2:(Shashank)
>   - rebased on amd-drm-staging-next
>   - changed amdgpu_seq64_gpu_addr
>
> Cc: Christian König 
> Cc: Alex Deucher 
> Cc: Felix Kuehling 
> Cc: Rajneesh Bhardwaj 
> Signed-off-by: Christian König 

Shashank can I get an rb on this patch here so that I can push it to
amd-staging-drm-next?

The patch was basically just to double check if the seq64 code works as
intended.

Thanks,
Christian.

> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c   | 14 
>   drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.h   |  1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 79 -
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  | 27 ++-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm_cpu.c  |  3 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c   |  2 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm_sdma.c |  5 ++
>   7 files changed, 42 insertions(+), 89 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c
> index 3d0d56087d41..300dc79fa2b9 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c
> @@ -199,6 +199,20 @@ void amdgpu_seq64_free(struct amdgpu_device *adev, u64 
> va)
>__clear_bit(bit_pos, adev->seq64.used);
>   }
>
> +/**
> + * amdgpu_seq64_gpu_addr - Calculate GPU addr from va
> + *
> + * @adev: amdgpu_device pointer
> + * @va: virtual address in client address space
> + *
> + * Calculate the GART address for a VA.
> + */
> +u64 amdgpu_seq64_gpu_addr(struct amdgpu_device *adev, u64 va)
> +{
> + return va - amdgpu_seq64_get_va_base(adev) +
> + amdgpu_bo_gpu_offset(adev->seq64.sbo);
> +}
> +
>   /**
>* amdgpu_seq64_fini - Cleanup seq64 driver
>*
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.h 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.h
> index 4203b2ab318d..63e8ac0a2057 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.h
> @@ -43,6 +43,7 @@ void amdgpu_seq64_free(struct amdgpu_device *adev, u64 
> gpu_addr);
>   int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm,
> struct amdgpu_bo_va **bo_va);
>   void amdgpu_seq64_unmap(struct amdgpu_device *adev, struct amdgpu_fpriv 
> *fpriv);
> +u64 amdgpu_seq64_gpu_addr(struct amdgpu_device *adev, u64 va);
>
>   #endif
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index ed4a8c5d26d7..0960e0a665d3 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -111,21 +111,6 @@ struct amdgpu_prt_cb {
>struct dma_fence_cb cb;
>   };
>
> -/**
> - * struct amdgpu_vm_tlb_seq_struct - Helper to increment the TLB flush 
> sequence
> - */
> -struct amdgpu_vm_tlb_seq_struct {
> - /**
> -  * @vm: pointer to the amdgpu_vm structure to set the fence sequence on
> -  */
> - struct amdgpu_vm *vm;
> -
> - /**
> -  * @cb: callback
> -  */
> - struct dma_fence_cb cb;
> -};
> -
>   /**
>* amdgpu_vm_set_pasid - manage pasid and vm ptr mapping
>*
> @@ -862,23 +847,6 @@ int amdgpu_vm_update_pdes(struct amdgpu_device *adev,
>return r;
>   }
>
> -/**
> - * amdgpu_vm_tlb_seq_cb - make sure to increment tlb sequence
> - * @fence: unused
> - * @cb: the callback structure
> - *
> - * Increments the tlb sequence to make sure that future CS execute a VM 
> flush.
> - */
> -static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
> -  struct dma_fence_cb *cb)
> -{
> - struct amdgpu_vm_tlb_seq_struct *tlb_cb;
> -
> - tlb_cb = container_of(cb, typeof(*tlb_cb), cb);
> - atomic64_inc(_cb->vm->tlb_seq);
> - kfree(tlb_cb);
> -}
> -
>   /**
>* amdgpu_vm_update_range - update a range in the vm page table
>*
> @@ -911,7 +879,6 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, 
> struct 

Re: [PATCH v2 3/3] drm/amdgpu: sync page table freeing with tlb flush

[Sharma, Shashank]

Hey Christian,

On 01/02/2024 14:48, Christian König wrote:



Am 31.01.24 um 18:14 schrieb Shashank Sharma:

This patch:
- Attaches the TLB flush fence to the PT objects being freed
- Adds a new ptr in VM to save this last TLB flush fence
- Adds a new lock in VM to prevent out-of-context update of saved
   TLB flush fence
- Adds a new ptr in tlb_flush structure to save vm

The idea is to delay freeing of page table objects until we have the
respective TLB entries flushed.

V2: rebase

Cc: Christian König 
Cc: Alex Deucher 
Cc: Felix Kuehling 
Cc: Rajneesh Bhardwaj 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    |  3 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |  4 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c | 27 +++
  .../gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c  | 13 +++--
  4 files changed, 45 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

index 67c690044b97..b0e81c249e3a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -2245,6 +2245,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, 
struct amdgpu_vm *vm,

  vm->generation = 0;
    mutex_init(>eviction_lock);
+    mutex_init(>tlb_flush_lock);
  vm->evicting = false;
  vm->tlb_fence_context = dma_fence_context_alloc(1);
  @@ -2360,7 +2361,9 @@ int amdgpu_vm_make_compute(struct 
amdgpu_device *adev, struct amdgpu_vm *vm)

  }
    dma_fence_put(vm->last_update);
+    dma_fence_put(vm->tlb_fence_last);
  vm->last_update = dma_fence_get_stub();
+    vm->tlb_fence_last = dma_fence_get_stub();
  vm->is_compute_context = true;
    /* Free the shadow bo for compute VM */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h

index 8e6fd25d07b7..b05bc586237f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -334,6 +334,10 @@ struct amdgpu_vm {
  uint64_t    *tlb_seq_cpu_addr;
  uint64_t    tlb_fence_context;
  +    /* Ptr and lock to maintain tlb flush sync */
+    struct mutex    tlb_flush_lock;
+    struct dma_fence    *tlb_fence_last;
+
  atomic64_t    kfd_last_flushed_seq;
    /* How many times we had to re-generate the page tables */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c

index 95dc0afdaffb..f1c4418c4d63 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
@@ -631,6 +631,18 @@ static int amdgpu_vm_pt_alloc(struct 
amdgpu_device *adev,

  return r;
  }
  +static inline
+void amdgpu_vm_attach_tlb_fence(struct amdgpu_bo *bo, struct 
dma_fence *fence)

+{
+    if (!bo || !fence)
+    return;
+
+    if (!dma_resv_reserve_fences(bo->tbo.base.resv, 1)) {
+    dma_resv_add_fence(bo->tbo.base.resv, fence,
+   DMA_RESV_USAGE_BOOKKEEP);
+    }
+}
+
  /**
   * amdgpu_vm_pt_free - free one PD/PT
   *
@@ -638,6 +650,7 @@ static int amdgpu_vm_pt_alloc(struct 
amdgpu_device *adev,

   */
  static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry)
  {
+    struct amdgpu_vm *vm;
  struct amdgpu_bo *shadow;
    if (!entry->bo)
@@ -646,9 +659,23 @@ static void amdgpu_vm_pt_free(struct 
amdgpu_vm_bo_base *entry)

  entry->bo->vm_bo = NULL;
  shadow = amdgpu_bo_shadowed(entry->bo);
  if (shadow) {
+    vm = shadow->vm_bo->vm;
+
+    mutex_lock(>tlb_flush_lock);
+    if (vm->tlb_fence_last)
+    amdgpu_vm_attach_tlb_fence(shadow, vm->tlb_fence_last);
+    mutex_unlock(>tlb_flush_lock);
+
  ttm_bo_set_bulk_move(>tbo, NULL);
  amdgpu_bo_unref();
  }
+
+    vm = entry->vm;
+    mutex_lock(>tlb_flush_lock);
+    if (vm->tlb_fence_last)
+    amdgpu_vm_attach_tlb_fence(entry->bo, vm->tlb_fence_last);
+    mutex_unlock(>tlb_flush_lock);
+


That approach doesn't make sense. Instead add the freed PT/PDs to a 
linked list in the parameters structure and only really free them 
after adding the fence to the root PD.


Sure, I will do those changes.

Just for the curiosity, why wouldn't this approach work ? Wouldn't this 
delay the actual freeing of buffers TTM until the fence signal ?


- Shashank





ttm_bo_set_bulk_move(>bo->tbo, NULL);
    spin_lock(>vm->status_lock);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c

index 569681badd7c..54ec81d30034 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_tlb_fence.c
@@ -31,6 +31,7 @@
  struct amdgpu_tlb_fence {
  struct dma_fence    base;
  struct amdgpu_device    *adev;
+    struct amdgpu_vm    *vm;


Big NAK to that. The VM might not live long enough to see the end of 
the TLB flush.


Regards,
Christian.


  struct dma_fence    

RE: [PATCH] drm/amdgpu: fix SI failure due to doorbells allocation

[Sharma, Shashank]

[AMD Official Use Only - General]

Reviewed-by: Shashank Sharma 

Regards
Shashank
-Original Message-
From: Icenowy Zheng 
Sent: Sunday, October 8, 2023 8:47 AM
To: Deucher, Alexander ; Koenig, Christian 
; Pan, Xinhui ; David Airlie 
; Daniel Vetter ; Sharma, Shashank 
; Yadav, Arvind 
Cc: amd-gfx@lists.freedesktop.org; dri-de...@lists.freedesktop.org; 
linux-ker...@vger.kernel.org; Icenowy Zheng 
Subject: [PATCH] drm/amdgpu: fix SI failure due to doorbells allocation

SI hardware does not have doorbells at all, however currently the code will try 
to do the allocation and thus fail, makes SI AMDGPU not usable.

Fix this failure by skipping doorbells allocation when doorbells count is zero.

Fixes: 54c30d2a8def ("drm/amdgpu: create kernel doorbell pages")
Signed-off-by: Icenowy Zheng 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c | 4 
 1 file changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
index d0249ada91d30..599aece42017a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
@@ -142,6 +142,10 @@ int amdgpu_doorbell_create_kernel_doorbells(struct 
amdgpu_device *adev)
int r;
int size;

+   /* SI HW does not have doorbells, skip allocation */
+   if (adev->doorbell.num_kernel_doorbells == 0)
+   return 0;
+
/* Reserve first num_kernel_doorbells (page-aligned) for kernel ops */
size = ALIGN(adev->doorbell.num_kernel_doorbells * sizeof(u32), 
PAGE_SIZE);

--
2.39.1

RE: [PATCH v6 3/9] drm/amdgpu: add new IOCTL for usermode queue

[Sharma, Shashank]

[AMD Official Use Only - General]

-Original Message-
From: Zhang, Yifan 
Sent: Wednesday, September 20, 2023 4:48 PM
To: Sharma, Shashank ; amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander ; Koenig, Christian 
; Yadav, Arvind ; Sharma, 
Shashank 
Subject: RE: [PATCH v6 3/9] drm/amdgpu: add new IOCTL for usermode queue

[AMD Official Use Only - General]

-Original Message-
From: amd-gfx  On Behalf Of Shashank 
Sharma
Sent: Saturday, September 9, 2023 12:05 AM
To: amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander ; Koenig, Christian 
; Yadav, Arvind ; Sharma, 
Shashank 
Subject: [PATCH v6 3/9] drm/amdgpu: add new IOCTL for usermode queue

This patch adds:
- A new IOCTL function to create and destroy
- A new structure to keep all the user queue data in one place.
- A function to generate unique index for the queue.

V1: Worked on review comments from RFC patch series:
  - Alex: Keep a list of queues, instead of single queue per process.
  - Christian: Use the queue manager instead of global ptrs,
   Don't keep the queue structure in amdgpu_ctx

V2: Worked on review comments:
 - Christian:
   - Formatting of text
   - There is no need for queuing of userqueues, with idr in place
 - Alex:
   - Remove use_doorbell, its unnecessary
   - Reuse amdgpu_mqd_props for saving mqd fields

 - Code formatting and re-arrangement

V3:
 - Integration with doorbell manager

V4:
 - Accommodate MQD union related changes in UAPI (Alex)
 - Do not set the queue size twice (Bas)

V5:
- Remove wrapper functions for queue indexing (Christian)
- Do not save the queue id/idr in queue itself (Christian)
- Move the idr allocation in the IP independent generic space
  (Christian)

V6:
- Check the validity of input IP type (Christian)

Cc: Alex Deucher 
Cc: Christian Koenig 
Signed-off-by: Shashank Sharma 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c   |   1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c | 116 ++
 .../gpu/drm/amd/include/amdgpu_userqueue.h|   2 +
 3 files changed, 119 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 2a89e303c3db..214a66b33dfa 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -2830,6 +2830,7 @@ const struct drm_ioctl_desc amdgpu_ioctls_kms[] = {
DRM_IOCTL_DEF_DRV(AMDGPU_GEM_VA, amdgpu_gem_va_ioctl, 
DRM_AUTH|DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(AMDGPU_GEM_OP, amdgpu_gem_op_ioctl, 
DRM_AUTH|DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(AMDGPU_GEM_USERPTR, amdgpu_gem_userptr_ioctl, 
DRM_AUTH|DRM_RENDER_ALLOW),
+   DRM_IOCTL_DEF_DRV(AMDGPU_USERQ, amdgpu_userq_ioctl,
+DRM_AUTH|DRM_RENDER_ALLOW),
 };

 static const struct drm_driver amdgpu_kms_driver = { diff --git 
a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
index effc0c7c02cf..44769423ba30 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
@@ -23,6 +23,122 @@
  */

 #include "amdgpu.h"
+#include "amdgpu_vm.h"
+#include "amdgpu_userqueue.h"
+
+static struct amdgpu_usermode_queue *
+amdgpu_userqueue_find(struct amdgpu_userq_mgr *uq_mgr, int qid) {
+   return idr_find(_mgr->userq_idr, qid); }
+
+static int
+amdgpu_userqueue_destroy(struct drm_file *filp, int queue_id) {
+   struct amdgpu_fpriv *fpriv = filp->driver_priv;
+   struct amdgpu_userq_mgr *uq_mgr = >userq_mgr;
+   const struct amdgpu_userq_funcs *uq_funcs;
+   struct amdgpu_usermode_queue *queue;
+
+   mutex_lock(_mgr->userq_mutex);
+
+   queue = amdgpu_userqueue_find(uq_mgr, queue_id);
+   if (!queue) {
+   DRM_DEBUG_DRIVER("Invalid queue id to destroy\n");
+   mutex_unlock(_mgr->userq_mutex);
+   return -EINVAL;
+   }
+   uq_funcs = uq_mgr->userq_funcs[queue->queue_type];
+   uq_funcs->mqd_destroy(uq_mgr, queue);
+   idr_remove(_mgr->userq_idr, queue_id);
+   kfree(queue);
+
+   mutex_unlock(_mgr->userq_mutex);
+   return 0;
+}
+
+static int
+amdgpu_userqueue_create(struct drm_file *filp, union drm_amdgpu_userq
+*args) {
+   struct amdgpu_fpriv *fpriv = filp->driver_priv;
+   struct amdgpu_userq_mgr *uq_mgr = >userq_mgr;
+   const struct amdgpu_userq_funcs *uq_funcs;
+   struct amdgpu_usermode_queue *queue;
+   int qid, r = 0;
+
+   /* Usermode queues are only supported for GFX/SDMA engines as of now */
+   if (args->in.ip_type != AMDGPU_HW_IP_GFX && args->in.ip_type != 
AMDGPU_HW_IP_DMA) {
+   DRM_ERROR("Usermode queue doesn't support IP type %u\n", 
args->in.ip_type);
+   return -EINVAL;
+   }
+
+   mutex_lock(_mgr->userq_mutex);
+
+   uq_funcs = uq_mgr->userq_funcs[args->in.ip_type];
+   if (!uq_funcs) {
+  

RE: [PATCH v6 1/5] drm/amdgpu: Allocate coredump memory in a nonblocking way

[Sharma, Shashank]

[AMD Official Use Only - General]

Pushed the rest of the patches in the series to amd-staging-drm-next.

Regards
Shashank
-Original Message-
From: Koenig, Christian 
Sent: Monday, September 11, 2023 1:15 PM
To: André Almeida ; dri-de...@lists.freedesktop.org; 
amd-gfx@lists.freedesktop.org; linux-ker...@vger.kernel.org; Sharma, Shashank 

Cc: kernel-...@igalia.com; Deucher, Alexander ; 
Pelloux-Prayer, Pierre-Eric 
Subject: Re: [PATCH v6 1/5] drm/amdgpu: Allocate coredump memory in a 
nonblocking way

Am 11.09.23 um 05:00 schrieb André Almeida:
> During a GPU reset, a normal memory reclaim could block to reclaim
> memory. Giving that coredump is a best effort mechanism, it shouldn't
> disturb the reset path. Change its memory allocation flag to a
> nonblocking one.

Since it is a bug fix I've already pushed this one into our internal branch 
quite a while ago.

Shashank can you take care of picking up the remaining patches and pushing them 
to amd-staging-drm-next?

Thanks,
Christian.

>
> Signed-off-by: André Almeida 
> Reviewed-by: Christian König 
> ---
> v5: no change
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index aa171db68639..bf4781551f88 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -4847,7 +4847,7 @@ static void amdgpu_reset_capture_coredumpm(struct 
> amdgpu_device *adev)
>   struct drm_device *dev = adev_to_drm(adev);
>
>   ktime_get_ts64(>reset_time);
> - dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_KERNEL,
> + dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_NOWAIT,
> amdgpu_devcoredump_read, amdgpu_devcoredump_free);
>   }
>   #endif

RE: [PATCH v6 1/5] drm/amdgpu: Allocate coredump memory in a nonblocking way

[Sharma, Shashank]

[AMD Official Use Only - General]

Hey Christian,

Will do that.

Regards
Shashank

-Original Message-
From: Koenig, Christian 
Sent: Monday, September 11, 2023 1:15 PM
To: André Almeida ; dri-de...@lists.freedesktop.org; 
amd-gfx@lists.freedesktop.org; linux-ker...@vger.kernel.org; Sharma, Shashank 

Cc: kernel-...@igalia.com; Deucher, Alexander ; 
Pelloux-Prayer, Pierre-Eric 
Subject: Re: [PATCH v6 1/5] drm/amdgpu: Allocate coredump memory in a 
nonblocking way

Am 11.09.23 um 05:00 schrieb André Almeida:
> During a GPU reset, a normal memory reclaim could block to reclaim
> memory. Giving that coredump is a best effort mechanism, it shouldn't
> disturb the reset path. Change its memory allocation flag to a
> nonblocking one.

Since it is a bug fix I've already pushed this one into our internal branch 
quite a while ago.

Shashank can you take care of picking up the remaining patches and pushing them 
to amd-staging-drm-next?

Thanks,
Christian.

>
> Signed-off-by: André Almeida 
> Reviewed-by: Christian König 
> ---
> v5: no change
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index aa171db68639..bf4781551f88 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -4847,7 +4847,7 @@ static void amdgpu_reset_capture_coredumpm(struct 
> amdgpu_device *adev)
>   struct drm_device *dev = adev_to_drm(adev);
>
>   ktime_get_ts64(>reset_time);
> - dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_KERNEL,
> + dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_NOWAIT,
> amdgpu_devcoredump_read, amdgpu_devcoredump_free);
>   }
>   #endif

RE: [PATCH 11/20] drm/amd/amdgpu/amdgpu_doorbell_mgr: Correct misdocumented param 'doorbell_index'

[Sharma, Shashank]

[AMD Official Use Only - General]

Reviewed-by: : Shashank Sharma 

Regards
Shashank
-Original Message-
From: Lee Jones 
Sent: Thursday, August 24, 2023 9:37 AM
To: l...@kernel.org
Cc: linux-ker...@vger.kernel.org; Deucher, Alexander 
; Koenig, Christian ; Pan, 
Xinhui ; David Airlie ; Daniel Vetter 
; Sharma, Shashank ; 
amd-gfx@lists.freedesktop.org; dri-de...@lists.freedesktop.org
Subject: [PATCH 11/20] drm/amd/amdgpu/amdgpu_doorbell_mgr: Correct 
misdocumented param 'doorbell_index'

Fixes the following W=1 kernel build warning(s):

 drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c:123: warning: Function 
parameter or member 'doorbell_index' not described in 
'amdgpu_doorbell_index_on_bar'
 drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c:123: warning: Excess function 
parameter 'db_index' description in 'amdgpu_doorbell_index_on_bar'

Signed-off-by: Lee Jones 
---
Cc: Alex Deucher 
Cc: "Christian König" 
Cc: "Pan, Xinhui" 
Cc: David Airlie 
Cc: Daniel Vetter 
Cc: Shashank Sharma 
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-de...@lists.freedesktop.org
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
index da4be0bbb4466..d0249ada91d30 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
@@ -113,7 +113,7 @@ void amdgpu_mm_wdoorbell64(struct amdgpu_device *adev, u32 
index, u64 v)
  *
  * @adev: amdgpu_device pointer
  * @db_bo: doorbell object's bo
- * @db_index: doorbell relative index in this doorbell object
+ * @doorbell_index: doorbell relative index in this doorbell object
  *
  * returns doorbell's absolute index in BAR
  */
--
2.42.0.rc1.204.g551eb34607-goog

RE: [PATCH] Revert "drm/amdgpu: don't modify num_doorbells for mes"

[Sharma, Shashank]

[AMD Official Use Only - General]

Nack to the revert.

This is a whole series of doorbell changes, which has replacement for the 
patches too.
https://patchwork.freedesktop.org/series/115802/

The whole series was pushed to staging branch, but looks like some of the 
patches did not merge.
When the whole series is merged, functionality will be restored.

I will check if there is any problem due to which other patches are blocked 
merge.

Regards
Shashank
-Original Message-
From: Zhang, Yifan 
Sent: Friday, August 4, 2023 7:01 AM
To: amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander ; Koenig, Christian 
; Sharma, Shashank ; Zhang, 
Yifan 
Subject: [PATCH] Revert "drm/amdgpu: don't modify num_doorbells for mes"

This reverts commit f46644aa8de6d5efeff8d8c7fbf3ed58a89c765c.

THe doorbell index could go beyond the first page for mes queues, this patch 
breaks the mes self test on gfx11.

[   23.212740] [drm] ring gfx_32768.1.1 was added
[   23.213147] [drm] ring compute_32768.2.2 was added
[   23.213540] [drm] ring sdma_32768.3.3 was added
[   23.213546] [drm:amdgpu_mm_wdoorbell64 [amdgpu]] *ERROR* writing beyond 
doorbell aperture: 0x1000!
[   23.214148] amdgpu :c2:00.0: amdgpu: gfx_v11_0_ring_set_wptr_gfx: 5168 
0x402 0x1000 100
[   23.560357] amdgpu :c2:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] 
*ERROR* ring gfx_32768.1.1 test failed (-110)

Signed-off-by: Yifan Zhang 
---
 .../gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c  | 34 +++
 1 file changed, 19 insertions(+), 15 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
index 5c0d3cea817d..31db526d4921 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
@@ -140,21 +140,25 @@ int amdgpu_doorbell_init(struct amdgpu_device *adev)
adev->doorbell.base = pci_resource_start(adev->pdev, 2);
adev->doorbell.size = pci_resource_len(adev->pdev, 2);

-   adev->doorbell.num_kernel_doorbells =
-   min_t(u32, adev->doorbell.size / sizeof(u32),
- adev->doorbell_index.max_assignment + 1);
-   if (adev->doorbell.num_kernel_doorbells == 0)
-   return -EINVAL;
-
-   /*
-* For Vega, reserve and map two pages on doorbell BAR since SDMA
-* paging queue doorbell use the second page. The
-* AMDGPU_DOORBELL64_MAX_ASSIGNMENT definition assumes all the
-* doorbells are in the first page. So with paging queue enabled,
-* the max num_kernel_doorbells should + 1 page (0x400 in dword)
-*/
-   if (adev->asic_type >= CHIP_VEGA10)
-   adev->doorbell.num_kernel_doorbells += 0x400;
+   if (adev->enable_mes) {
+   adev->doorbell.num_kernel_doorbells =
+   adev->doorbell.size / sizeof(u32);
+   } else {
+   adev->doorbell.num_kernel_doorbells =
+   min_t(u32, adev->doorbell.size / sizeof(u32),
+ adev->doorbell_index.max_assignment+1);
+   if (adev->doorbell.num_kernel_doorbells == 0)
+   return -EINVAL;
+
+   /* For Vega, reserve and map two pages on doorbell BAR since 
SDMA
+* paging queue doorbell use the second page. The
+* AMDGPU_DOORBELL64_MAX_ASSIGNMENT definition assumes all the
+* doorbells are in the first page. So with paging queue 
enabled,
+* the max num_kernel_doorbells should + 1 page (0x400 in dword)
+*/
+   if (adev->asic_type >= CHIP_VEGA10)
+   adev->doorbell.num_kernel_doorbells += 0x400;
+   }

adev->doorbell.ptr = ioremap(adev->doorbell.base,
 adev->doorbell.num_kernel_doorbells *
--
2.37.3

RE: [PATCH v4 05/10] drm/amdgpu: create context space for usermode queue

[Sharma, Shashank]

[Public]

> CSA, GDS backup, and shadow are allocated by userspace now.

Noted Alex, thanks. I will update the patch series and userspace accordingly. 

Regards
Shashank

-Original Message-
From: Deucher, Alexander  
Sent: 25 April 2023 19:38
To: Sharma, Shashank ; Koenig, Christian 
; amd-gfx@lists.freedesktop.org
Cc: Yadav, Arvind ; Pelloux-Prayer, Pierre-Eric 
; contactshashanksha...@gmail.com
Subject: RE: [PATCH v4 05/10] drm/amdgpu: create context space for usermode 
queue

[Public]

> -Original Message-
> From: Sharma, Shashank 
> Sent: Tuesday, April 25, 2023 9:13 AM
> To: Koenig, Christian ; amd- 
> g...@lists.freedesktop.org
> Cc: Yadav, Arvind ; Pelloux-Prayer, Pierre-Eric 
> ;
> contactshashanksha...@gmail.com; Deucher, Alexander 
> 
> Subject: Re: [PATCH v4 05/10] drm/amdgpu: create context space for 
> usermode queue
> 
> 
> On 25/04/2023 14:30, Christian König wrote:
> > Am 24.04.23 um 19:38 schrieb Shashank Sharma:
> >> The FW expects us to allocate at least one page as context space to 
> >> process gang, process, GDS and FW  related work.
> >> This patch creates a joint object for the same, and calculates GPU 
> >> space offsets for each of these spaces.
> >>
> >> V1: Addressed review comments on RFC patch:
> >>  Alex: Make this function IP specific
> >>
> >> V2: Addressed review comments from Christian
> >>  - Allocate only one object for total FW space, and calculate
> >>    offsets for each of these objects.
> >>
> >> V3: Integration with doorbell manager
> >> V4: Review comments:
> >>  - Remove shadow from FW space list from cover letter (Alex)
> >>  - Alignment of macro (Luben)
> >>
> >> Cc: Alex Deucher 
> >> Cc: Christian Koenig 
> >> Signed-off-by: Shashank Sharma 
> >> ---
> >>   drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c    | 57
> >> ++-
> >>   .../gpu/drm/amd/include/amdgpu_userqueue.h    |  6 ++
> >>   2 files changed, 61 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
> >> b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
> >> index 9f7b14966ac8..f6b33faea86f 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
> >> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
> >> @@ -53,6 +53,11 @@
> >>   #define GFX11_NUM_GFX_RINGS    1
> >>   #define GFX11_MEC_HPD_SIZE    2048
> >>   +#define AMDGPU_USERQ_PROC_CTX_SZ   PAGE_SIZE
> >> +#define AMDGPU_USERQ_GANG_CTX_SZ   PAGE_SIZE #define 
> >> +AMDGPU_USERQ_FW_CTX_SZ PAGE_SIZE #define
> AMDGPU_USERQ_GDS_CTX_SZ
> >> +PAGE_SIZE
> >> +
> >>   #define RLCG_UCODE_LOADING_START_ADDRESS    0x2000L
> >>   #define RLC_PG_DELAY_3_DEFAULT_GC_11_0_1    0x1388
> >>   @@ -6406,6 +6411,44 @@ const struct amdgpu_ip_block_version 
> >> gfx_v11_0_ip_block =
> >>   .funcs = _v11_0_ip_funcs,
> >>   };
> >>   +static int gfx_v11_userq_create_ctx_space(struct 
> >> amdgpu_userq_mgr *uq_mgr,
> >> +  struct amdgpu_usermode_queue *queue) {
> >> +    struct amdgpu_device *adev = uq_mgr->adev;
> >> +    struct amdgpu_userq_ctx_space *ctx = >fw_space;
> >> +    int r, size;
> >> +
> >> +    /*
> >> + * The FW expects at least one page space allocated for
> >> + * process ctx, gang ctx, gds ctx, fw ctx each.
> >> + */
> >> +    size = AMDGPU_USERQ_PROC_CTX_SZ +
> AMDGPU_USERQ_FW_CTX_SZ +
> >> +   AMDGPU_USERQ_GANG_CTX_SZ +
> AMDGPU_USERQ_GDS_CTX_SZ;
> >> +    r = amdgpu_bo_create_kernel(adev, size, PAGE_SIZE,
> >> +    AMDGPU_GEM_DOMAIN_GTT,
> >> +    >obj,
> >> +    >gpu_addr,
> >> +    >cpu_ptr);
> >
> > Wasn't this stuff provided by userspace now?
> >
> The last I checked, only the shadow buffer was coming from userspace, 
> and we were thinking about other stuff. I might not be well updated if 
> there any recent development here with the short term solution.

CSA, GDS backup, and shadow are allocated by userspace now.

Alex

> 
> - Shashank
> 
> > Christian.
> >
> >> +    if (r) {
> >> +    DRM_ERROR("Failed to allocate ctx space bo for userqueue,
> >> err:%d\n", r);
> >> +    return r;
> >> +    }
> >> +
> >> +    queue->proc_ctx_gpu_addr = ctx->gpu_addr;
> >> +    queue->gang_

RE: [PATCH v3 0/9] AMDGPU Usermode queues

[Sharma, Shashank]

[AMD Official Use Only - General]

Hello Bas, 

This is not the correct interpretation of the code, the USERQ_IOCTL has both 
the OPs (create and destroy), but th euser has to exclusively call  it.

Please see the sample test program in the existing libDRM series (userq_test.c, 
it specifically calls amdgpu_free_userq, which does the destroy_OP

for the IOCTL.

- Shashank

-Original Message-
From: Bas Nieuwenhuizen  
Sent: 10 April 2023 11:26
To: Sharma, Shashank 
Cc: amd-gfx@lists.freedesktop.org; Deucher, Alexander 
; Kuehling, Felix ; Koenig, 
Christian ; Yadav, Arvind 
Subject: Re: [PATCH v3 0/9] AMDGPU Usermode queues

Hi Shashank,

I think I found the issue: I wasn't destroying the user queue in my program and 
the kernel doesn't clean up any remaining user queues in the postclose hook. I 
think we need something like
https://github.com/BNieuwenhuizen/linux/commit/e90c8d1185da7353c12837973ceddf55ccc85d29
?

While running things multiple times now works, I still have problems doing 
multiple submissions from the same queue. Looking forward to the updated 
test/sample

Thanks,
Bas

On Mon, Apr 10, 2023 at 9:32 AM Sharma, Shashank  
wrote:
>
> [AMD Official Use Only - General]
>
> Hello Bas,
> Thanks for trying this out.
>
> This could be due to the doorbell as you mentioned, Usermode queue uses 
> doorbell manager internally.
> This week, we are planning to publis the latest libDRM sample code which uses 
> a doorbell object (instead of the doorbell hack IOCTL), adapting to that 
> should fix your problem in my opinion.
> We have tested this full stack (libDRM test + Usermode queue + doorbell 
> manager) for 500+ consecutive runs, and it worked well for us.
>
> You can use this integrated kernel stack (1+2) from my gitlab to build 
> your kernel: 
> https://gitlab.freedesktop.org/contactshashanksharma/userq-amdgpu/-/tr
> ee/integrated-db-and-uq-v3 Please stay tuned for updated libDRM 
> changes with doorbell objects.
>
> Regards
> Shashank
> -Original Message-
> From: Bas Nieuwenhuizen 
> Sent: 10 April 2023 02:37
> To: Sharma, Shashank 
> Cc: amd-gfx@lists.freedesktop.org; Deucher, Alexander 
> ; Kuehling, Felix ; 
> Koenig, Christian 
> Subject: Re: [PATCH v3 0/9] AMDGPU Usermode queues
>
> Hi Shashank,
>
> I tried writing a program to experiment with usermode queues and I found some 
> weird behavior: The first run of the program works as expected, while 
> subsequent runs don't seem to do anything (and I allocate everything in GTT, 
> so it should be zero initialized consistently). Is this a known issue?
>
> The linked libdrm code for the uapi still does a doorbell ioctl so it could 
> very well be that I do the doorbell wrong (especially since the ioctl 
> implementation was never shared AFAICT?), but it seems like the kernel 
> submissions (i.e. write wptr in dwords to the wptr va and to the doorbell). 
> Is it possible to update the test in libdrm?
>
> Code: https://gitlab.freedesktop.org/bnieuwenhuizen/usermode-queue
>
> Thanks,
> Bas
>
> On Wed, Mar 29, 2023 at 6:05 PM Shashank Sharma  
> wrote:
> >
> > This patch series introduces AMDGPU usermode queues for gfx workloads.
> > Usermode queues is a method of GPU workload submission into the 
> > graphics hardware without any interaction with kernel/DRM schedulers.
> > In this method, a userspace graphics application can create its own 
> > workqueue and submit it directly in the GPU HW.
> >
> > The general idea of how this is supposed to work:
> > - The application creates the following GPU objetcs:
> >   - A queue object to hold the workload packets.
> >   - A read pointer object.
> >   - A write pointer object.
> >   - A doorbell page.
> > - The application picks a 32-bit offset in the doorbell page for this queue.
> > - The application uses the usermode_queue_create IOCTL introduced in
> >   this patch, by passing the the GPU addresses of these objects (read
> >   ptr, write ptr, queue base address and 32-bit doorbell offset from
> >   the doorbell page)
> > - The kernel creates the queue and maps it in the HW.
> > - The application can start submitting the data in the queue as soon as
> >   the kernel IOCTL returns.
> > - After filling the workload data in the queue, the app must write the
> >   number of dwords added in the queue into the doorbell offset, and the
> >   GPU will start fetching the data.
> >
> > libDRM changes for this series and a sample DRM test program can be 
> > found in the MESA merge request here:
> > https://gitlab.freedesktop.org/mesa/drm/-/merge_requests/287
> >
> > This patch series depends on the doorbell-manager changes, which are 
> > being reviewed here:
>

RE: [PATCH v3 0/9] AMDGPU Usermode queues

[Sharma, Shashank]

[AMD Official Use Only - General]

Hello Bas, 
Thanks for trying this out. 

This could be due to the doorbell as you mentioned, Usermode queue uses 
doorbell manager internally.
This week, we are planning to publis the latest libDRM sample code which uses a 
doorbell object (instead of the doorbell hack IOCTL), adapting to that should 
fix your problem in my opinion. 
We have tested this full stack (libDRM test + Usermode queue + doorbell 
manager) for 500+ consecutive runs, and it worked well for us.

You can use this integrated kernel stack (1+2) from my gitlab to build your 
kernel: 
https://gitlab.freedesktop.org/contactshashanksharma/userq-amdgpu/-/tree/integrated-db-and-uq-v3
 
Please stay tuned for updated libDRM changes with doorbell objects.

Regards
Shashank
-Original Message-
From: Bas Nieuwenhuizen  
Sent: 10 April 2023 02:37
To: Sharma, Shashank 
Cc: amd-gfx@lists.freedesktop.org; Deucher, Alexander 
; Kuehling, Felix ; Koenig, 
Christian 
Subject: Re: [PATCH v3 0/9] AMDGPU Usermode queues

Hi Shashank,

I tried writing a program to experiment with usermode queues and I found some 
weird behavior: The first run of the program works as expected, while 
subsequent runs don't seem to do anything (and I allocate everything in GTT, so 
it should be zero initialized consistently). Is this a known issue?

The linked libdrm code for the uapi still does a doorbell ioctl so it could 
very well be that I do the doorbell wrong (especially since the ioctl 
implementation was never shared AFAICT?), but it seems like the kernel 
submissions (i.e. write wptr in dwords to the wptr va and to the doorbell). Is 
it possible to update the test in libdrm?

Code: https://gitlab.freedesktop.org/bnieuwenhuizen/usermode-queue

Thanks,
Bas

On Wed, Mar 29, 2023 at 6:05 PM Shashank Sharma  wrote:
>
> This patch series introduces AMDGPU usermode queues for gfx workloads.
> Usermode queues is a method of GPU workload submission into the 
> graphics hardware without any interaction with kernel/DRM schedulers. 
> In this method, a userspace graphics application can create its own 
> workqueue and submit it directly in the GPU HW.
>
> The general idea of how this is supposed to work:
> - The application creates the following GPU objetcs:
>   - A queue object to hold the workload packets.
>   - A read pointer object.
>   - A write pointer object.
>   - A doorbell page.
> - The application picks a 32-bit offset in the doorbell page for this queue.
> - The application uses the usermode_queue_create IOCTL introduced in
>   this patch, by passing the the GPU addresses of these objects (read
>   ptr, write ptr, queue base address and 32-bit doorbell offset from
>   the doorbell page)
> - The kernel creates the queue and maps it in the HW.
> - The application can start submitting the data in the queue as soon as
>   the kernel IOCTL returns.
> - After filling the workload data in the queue, the app must write the
>   number of dwords added in the queue into the doorbell offset, and the
>   GPU will start fetching the data.
>
> libDRM changes for this series and a sample DRM test program can be 
> found in the MESA merge request here:
> https://gitlab.freedesktop.org/mesa/drm/-/merge_requests/287
>
> This patch series depends on the doorbell-manager changes, which are 
> being reviewed here:
> https://patchwork.freedesktop.org/series/115802/
>
> Alex Deucher (1):
>   drm/amdgpu: UAPI for user queue management
>
> Arvind Yadav (2):
>   drm/amdgpu: add new parameters in v11_struct
>   drm/amdgpu: map wptr BO into GART
>
> Shashank Sharma (6):
>   drm/amdgpu: add usermode queue base code
>   drm/amdgpu: add new IOCTL for usermode queue
>   drm/amdgpu: create GFX-gen11 MQD for userqueue
>   drm/amdgpu: create context space for usermode queue
>   drm/amdgpu: map usermode queue into MES
>   drm/amdgpu: generate doorbell index for userqueue
>
>  drivers/gpu/drm/amd/amdgpu/Makefile   |   3 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h   |  10 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c   |   2 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c   |   6 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c | 298 
> ++  .../drm/amd/amdgpu/amdgpu_userqueue_gfx_v11.c | 230 
> ++
>  .../gpu/drm/amd/include/amdgpu_userqueue.h|  66 
>  drivers/gpu/drm/amd/include/v11_structs.h |  16 +-
>  include/uapi/drm/amdgpu_drm.h |  55 
>  9 files changed, 677 insertions(+), 9 deletions(-)  create mode 
> 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
>  create mode 100644 
> drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue_gfx_v11.c
>  create mode 100644 drivers/gpu/drm/amd/include/amdgpu_userqueue.h
>
> --
> 2.40.0
>

RE: [PATCH 00/16] AMDGPU Doorbell manager

[Sharma, Shashank]

[AMD Official Use Only - General]

Hey Luben, 

Agree and noted.
I have configured my editor to write the code according to the alignment 
conventions, but probably something missed the mark. 

- Shashank

-Original Message-
From: Tuikov, Luben  
Sent: 30 March 2023 15:50
To: Sharma, Shashank ; amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander ; Joshi, Mukul 
; Kuehling, Felix ; Koenig, 
Christian 
Subject: Re: [PATCH 00/16] AMDGPU Doorbell manager

As I'm reviewing this, it is obvious that this patchset hasn't gone though 
scripts/checkpatch.pl.

It's good practice to run one's patches through scripts/checkpatch.pl, to see 
deviations on common Linux practices, and correct them.

Regards,
Luben

On 2023-03-29 11:47, Shashank Sharma wrote:
> The doorbells in AMDGPU drivers are currently managed by different 
> users in a scattered way, across the driver. The existing clients are:
> - AMDGPU graphics driver for kernel level doorbell writes.
> - AMDGPU MES module for kernel level doorbell write (MES ring test).
> - AMDGPU MES modules for kernel level aggregated doorbell writes.
> - AMDGPU MES module for MES process doorbell writes.
> - AMDKFD module for KFD/KIQ kernel doorbell writes.
> - AMDKFD module for KFD process doorbell writes.
> - AMDGPU usermode queues for usermode doorbell writes (upcoming).
> 
> This patch series introduces Doorbell-manager to keep the doorbell 
> handling at a central place. The fundamental changes are:
> 
> - Introduce and accommodate a new GEM domain for doorbells.
> - Prepare the AMDGPU ttm backend for handling doorbell allocation.
> - Introduce doorbell-manager functions to allocate, free and index
>   doorbells in one unique way.
> - Create doorbell BOs for kernel-level and process level doorbell
>   opertations, and place it in existing structures.
> - Modify the existing graphics, KFD and MES code to use the
>   doorbell-manager functions.
> - Remove the existing doorbell management code in KFD/MES.
> 
> PS: This series has been sanity tested with kfd_test_suit to ensure
> it is not introducing any regressions due to kfd doorbell changes.
> 
> The idea is that:
> - a kernel client can call doorbell manager functions to allocate/free
>   doorbell pages.
> - a usermode app can directly allocate a page from the doorbell bar just
>   like a GEM object and use it for different usermode queues.
> 
> Alex Deucher (2):
>   drm/amdgpu: add UAPI for allocating doorbell memory
>   drm/amdgpu: accommodate DOMAIN/PL_DOORBELL
> 
> Shashank Sharma (14):
>   drm/amdgpu: rename num_doorbells
>   drm/amdgpu: include protection for doobell.h
>   drm/amdgpu: create a new file for doorbell manager
>   drm/amdgpu: don't modify num_doorbells for mes
>   drm/amdgpu: add helper to create doorbell pages
>   drm/amdgpu: initialize ttm for doorbells
>   drm/amdgpu: create kernel doorbell page
>   drm/amdgpu: validate doorbell read/write
>   drm/amdgpu: get absolute offset from doorbell index
>   drm/amdgpu: use doorbell manager for kfd kernel doorbells
>   drm/amdgpu: use doorbell manager for kfd process doorbells
>   drm/amdgpu: remove ununsed functions and variables
>   drm/amdgpu: use doorbell mgr for MES kernel doorbells
>   drm/amdgpu: user doorbell mgr for MES process doorbells
> 
>  drivers/gpu/drm/amd/amdgpu/Makefile   |   2 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c|   6 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c| 164 --
>  drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h  | 102 +-  
> .../gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c  | 304 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c   | 165 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h   |  17 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c|  11 +-
>  .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h|   2 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c   |  31 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h   |   1 +
>  drivers/gpu/drm/amd/amdkfd/kfd_chardev.c  |  13 -
>  drivers/gpu/drm/amd/amdkfd/kfd_device.c   |   4 +-
>  .../drm/amd/amdkfd/kfd_device_queue_manager.c |  16 +-
>  drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c | 198 
>  drivers/gpu/drm/amd/amdkfd/kfd_priv.h |  23 +-
>  drivers/gpu/drm/amd/amdkfd/kfd_process.c  |  26 +-
>  .../amd/amdkfd/kfd_process_queue_manager.c|  16 +-
>  include/uapi/drm/amdgpu_drm.h |   7 +-
>  19 files changed, 636 insertions(+), 472 deletions(-)  create mode 
> 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
> 

RE: [PATCH v3 1/5] drm/amdgpu: add UAPI for workload hints to ctx ioctl

[Sharma, Shashank]

[AMD Official Use Only - General]

From the exposed workload hints:
+#define AMDGPU_CTX_WORKLOAD_HINT_NONE
+#define AMDGPU_CTX_WORKLOAD_HINT_3D
+#define AMDGPU_CTX_WORKLOAD_HINT_VIDEO
+#define AMDGPU_CTX_WORKLOAD_HINT_VR
+#define AMDGPU_CTX_WORKLOAD_HINT_COMPUTE

I guess the only option which we do not know how to use is HINT_VR, everything 
else is known. I find it a limitation of the stack that we can’t differentiate 
between a VR workload Vs 3D, coz at some time we might have to give high 
privilege or special attention to it when VR becomes more demanding, but for 
now, I can remove this one option from the patch:

+#define AMDGPU_CTX_WORKLOAD_HINT_VR
Regards
Shashank

From: Koenig, Christian 
Sent: 22 March 2023 15:29
To: Marek Olšák 
Cc: Christian König ; Sharma, Shashank 
; Deucher, Alexander ; 
Somalapuram, Amaranath ; 
amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH v3 1/5] drm/amdgpu: add UAPI for workload hints to ctx ioctl

Well that sounds like being able to optionally set it after context creation is 
actually the right approach.

VA-API could set it as soon as we know that this is a video codec application.

Vulkan can set it depending on what features are used by the application.

But yes, Shashank (or whoever requested that) should come up with some code for 
Mesa to actually use it. Otherwise we don't have the justification to push it 
into the kernel driver.

Christian.
Am 22.03.23 um 15:24 schrieb Marek Olšák:
The hint is static per API (one of graphics, video, compute, unknown). In the 
case of Vulkan, which exposes all queues, the hint is unknown, so Vulkan won't 
use it. (or make it based on the queue being used and not the uapi context 
state) GL won't use it because the default hint is already 3D. That makes VAAPI 
the only user that only sets the hint once, and maybe it's not worth even 
adding this uapi just for VAAPI.

Marek

On Wed, Mar 22, 2023 at 10:08 AM Christian König 
mailto:christian.koe...@amd.com>> wrote:
Well completely agree that we shouldn't have unused API. That's why I said we 
should remove the getting the hint from the UAPI.

But what's wrong with setting it after creating the context? Don't you know 
enough about the use case? I need to understand the background a bit better 
here.

Christian.
Am 22.03.23 um 15:05 schrieb Marek Olšák:
The option to change the hint after context creation and get the hint would be 
unused uapi, and AFAIK we are not supposed to add unused uapi. What I asked is 
to change it to a uapi that userspace will actually use.

Marek

On Tue, Mar 21, 2023 at 9:54 AM Christian König 
mailto:ckoenig.leichtzumer...@gmail.com>> 
wrote:
Yes, I would like to avoid having multiple code paths for context creation.

Setting it later on should be equally to specifying it on creation since we 
only need it during CS.

Regards,
Christian.
Am 21.03.23 um 14:00 schrieb Sharma, Shashank:

[AMD Official Use Only - General]

When we started this patch series, the workload hint was a part of the ctx_flag 
only,
But we changed that after the design review, to make it more like how we are 
handling PSTATE.

Details:
https://patchwork.freedesktop.org/patch/496111/

Regards
Shashank

From: Marek Olšák <mailto:mar...@gmail.com>
Sent: 21 March 2023 04:05
To: Sharma, Shashank <mailto:shashank.sha...@amd.com>
Cc: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; 
Deucher, Alexander 
<mailto:alexander.deuc...@amd.com>; Somalapuram, 
Amaranath 
<mailto:amaranath.somalapu...@amd.com>; Koenig, 
Christian <mailto:christian.koe...@amd.com>
Subject: Re: [PATCH v3 1/5] drm/amdgpu: add UAPI for workload hints to ctx ioctl

I think we should do it differently because this interface will be mostly 
unused by open source userspace in its current form.

Let's set the workload hint in drm_amdgpu_ctx_in::flags, and that will be 
immutable for the lifetime of the context. No other interface is needed.

Marek

On Mon, Sep 26, 2022 at 5:41 PM Shashank Sharma 
mailto:shashank.sha...@amd.com>> wrote:
Allow the user to specify a workload hint to the kernel.
We can use these to tweak the dpm heuristics to better match
the workload for improved performance.

V3: Create only set() workload UAPI (Christian)

Signed-off-by: Alex Deucher 
mailto:alexander.deuc...@amd.com>>
Signed-off-by: Shashank Sharma 
mailto:shashank.sha...@amd.com>>
---
 include/uapi/drm/amdgpu_drm.h | 17 +
 1 file changed, 17 insertions(+)

diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
index c2c9c674a223..23d354242699 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -212,6 +212,7 @@ union drm_amdgpu_bo_list {
 #define AMDGPU_CTX_OP_QUERY_STATE2 4
 #define AMDGPU_CTX_OP_GET_STABLE_PSTATE5
 #define AMDGPU_CTX_OP_SET_STABLE_PSTATE6
+#define AMDGPU_CTX_OP_SET_WORKLOAD_PROFILE 7

 /* GPU reset status */
 #define AMDGPU_CTX_NO_RESET

RE: [PATCH v3 1/5] drm/amdgpu: add UAPI for workload hints to ctx ioctl

[Sharma, Shashank]

[AMD Official Use Only - General]

When we started this patch series, the workload hint was a part of the ctx_flag 
only,
But we changed that after the design review, to make it more like how we are 
handling PSTATE.

Details:
https://patchwork.freedesktop.org/patch/496111/

Regards
Shashank

From: Marek Olšák 
Sent: 21 March 2023 04:05
To: Sharma, Shashank 
Cc: amd-gfx@lists.freedesktop.org; Deucher, Alexander 
; Somalapuram, Amaranath 
; Koenig, Christian 
Subject: Re: [PATCH v3 1/5] drm/amdgpu: add UAPI for workload hints to ctx ioctl

I think we should do it differently because this interface will be mostly 
unused by open source userspace in its current form.

Let's set the workload hint in drm_amdgpu_ctx_in::flags, and that will be 
immutable for the lifetime of the context. No other interface is needed.

Marek

On Mon, Sep 26, 2022 at 5:41 PM Shashank Sharma 
mailto:shashank.sha...@amd.com>> wrote:
Allow the user to specify a workload hint to the kernel.
We can use these to tweak the dpm heuristics to better match
the workload for improved performance.

V3: Create only set() workload UAPI (Christian)

Signed-off-by: Alex Deucher 
mailto:alexander.deuc...@amd.com>>
Signed-off-by: Shashank Sharma 
mailto:shashank.sha...@amd.com>>
---
 include/uapi/drm/amdgpu_drm.h | 17 +
 1 file changed, 17 insertions(+)

diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
index c2c9c674a223..23d354242699 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -212,6 +212,7 @@ union drm_amdgpu_bo_list {
 #define AMDGPU_CTX_OP_QUERY_STATE2 4
 #define AMDGPU_CTX_OP_GET_STABLE_PSTATE5
 #define AMDGPU_CTX_OP_SET_STABLE_PSTATE6
+#define AMDGPU_CTX_OP_SET_WORKLOAD_PROFILE 7

 /* GPU reset status */
 #define AMDGPU_CTX_NO_RESET0
@@ -252,6 +253,17 @@ union drm_amdgpu_bo_list {
 #define AMDGPU_CTX_STABLE_PSTATE_MIN_MCLK  3
 #define AMDGPU_CTX_STABLE_PSTATE_PEAK  4

+/* GPU workload hints, flag bits 8-15 */
+#define AMDGPU_CTX_WORKLOAD_HINT_SHIFT 8
+#define AMDGPU_CTX_WORKLOAD_HINT_MASK  (0xff << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_NONE  (0 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_3D(1 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_VIDEO (2 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_VR(3 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_COMPUTE   (4 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_MAX  AMDGPU_CTX_WORKLOAD_HINT_COMPUTE
+#define AMDGPU_CTX_WORKLOAD_INDEX(n)  (n >> AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+
 struct drm_amdgpu_ctx_in {
/** AMDGPU_CTX_OP_* */
__u32   op;
@@ -281,6 +293,11 @@ union drm_amdgpu_ctx_out {
__u32   flags;
__u32   _pad;
} pstate;
+
+   struct {
+   __u32   flags;
+   __u32   _pad;
+   } workload;
 };

 union drm_amdgpu_ctx {
--
2.34.1

RE: [RFC PATCH v2 00/18] Add DRM CRTC 3D LUT interface

[Sharma, Shashank]

[AMD Official Use Only - General]

+ Uday, for awareness. 

Regards
Shashank
-Original Message-
From: Pekka Paalanen 
Sent: 14 February 2023 10:28
To: Melissa Wen 
Cc: Ville Syrjälä ; 
dri-de...@lists.freedesktop.org; airl...@gmail.com; 
laurent.pinchart+rene...@ideasonboard.com; Sharma, Shashank 
; Siqueira, Rodrigo ; 
amd-gfx@lists.freedesktop.org; Hung, Alex ; Wentland, Harry 
; tzimmerm...@suse.de; Li, Sun peng (Leo) 
; maarten.lankho...@linux.intel.com; mrip...@kernel.org; 
seanp...@chromium.org; dan...@ffwll.ch; Lakha, Bhawanpreet 
; Kim, Sung joon ; 
cont...@emersion.fr; Pan, Xinhui ; Koenig, Christian 
; kernel-...@igalia.com; Deucher, Alexander 
; Kazlauskas, Nicholas 
; Joshua Ashton 
Subject: Re: [RFC PATCH v2 00/18] Add DRM CRTC 3D LUT interface

On Mon, 13 Feb 2023 18:45:40 -0100
Melissa Wen  wrote:

> On 02/13, Ville Syrjälä wrote:
> > On Mon, Feb 13, 2023 at 11:01:31AM +0200, Pekka Paalanen wrote:  
> > > On Fri, 10 Feb 2023 14:47:50 -0500 Harry Wentland 
> > >  wrote:
> > >   
> > > > On 2/10/23 04:28, Pekka Paalanen wrote:  
> > > > > On Thu, 9 Feb 2023 13:27:02 -0100 Melissa Wen 
> > > > >  wrote:
> > > > > 
> > > > >> On 01/31, Pekka Paalanen wrote:
> > > > >>> On Mon, 9 Jan 2023 14:38:09 -0100 Melissa Wen 
> > > > >>>  wrote:
> > > > >>>   
> > > > >>>> On 01/09, Melissa Wen wrote:  
> > > > >>>>> Hi,
> > > > >>>>>
> > > > >>>>> After collecting comments in different places, here is a 
> > > > >>>>> second version of the work on adding DRM CRTC 3D LUT 
> > > > >>>>> support to the current DRM color mgmt interface. In 
> > > > >>>>> comparison to previous proposals [1][2][3], here we add 3D 
> > > > >>>>> LUT before gamma 1D LUT, but also a shaper 1D LUT before 3D LUT, 
> > > > >>>>> that means the following DRM CRTC color correction pipeline:
> > > > >>>>>
> > > > >>>>> Blend -> Degamma 1D LUT -> CTM -> Shaper 1D LUT -> 3D LUT -> 
> > > > >>>>> Gamma 1D LUT  
> > > 
> > > ...
> > >   
> > > > >>> +/*
> > > > >>> + * struct drm_mode_lut3d_mode - 3D LUT mode information.
> > > > >>> + * @lut_size: number of valid points on every dimension of 3D LUT.
> > > > >>> + * @lut_stride: number of points on every dimension of 3D LUT.
> > > > >>> + * @bit_depth: number of bits of RGB. If color_mode defines 
> > > > >>> entries with higher
> > > > >>> + * bit_depth the least significant bits will be 
> > > > >>> truncated.
> > > > >>> + * @color_format: fourcc values, ex. DRM_FORMAT_XRGB16161616 or 
> > > > >>> DRM_FORMAT_XBGR16161616.
> > > > >>> + * @flags: flags for hardware-sepcific features  */ struct 
> > > > >>> +drm_mode_lut3d_mode {
> > > > >>> +   __u16 lut_size;
> > > > >>> +   __u16 lut_stride[3];
> > > > >>> +   __u16 bit_depth;
> > > > >>> +   __u32 color_format;
> > > > >>> +   __u32 flags;
> > > > >>> +};
> > > 
> > > ...
> > >   
> > > > >>> What is "number of bits of RGB"? Input precision? Output precision?
> > > > >>> Integer or floating point?  
> > > > >>
> > > > >> It's the bit depth of the 3D LUT values, the same for every 
> > > > >> channels. In
> > > > >> the AMD case, it's supports 10-bit and 12-bit, for example.
> > > > > 
> > > > > Ok. So e.g. r5g6b5 is not a possible 3D LUT element type on 
> > > > > any hardware ever?
> > > > > 
> > > > 
> > > > I haven't had a chance to go through all patches yet but if this 
> > > > is modeled after Alex Hung's work this should be covered by 
> > > > color_format.
> > > > The idea is that color_format takes a FOURCC value and defines 
> > > > the format of the entries in the 3DLUT blob.
> > > > 
> > > > The bit_depth describes the actual bit depth that the HW supports.
> > > > E.g., color_format could be DRM_FORMAT_XRGB16161616 but HW might 
> > > 

Re: [PATCH v3 5/5] drm/amdgpu: switch workload context to/from compute

[Sharma, Shashank]

On 9/30/2022 11:54 AM, Lazar, Lijo wrote:



On 9/30/2022 2:52 PM, Sharma, Shashank wrote:



On 9/30/2022 11:13 AM, Lazar, Lijo wrote:



On 9/30/2022 2:07 PM, Sharma, Shashank wrote:



On 9/30/2022 7:08 AM, Lazar, Lijo wrote:



On 9/30/2022 12:02 AM, Alex Deucher wrote:
On Thu, Sep 29, 2022 at 10:14 AM Lazar, Lijo  
wrote:




On 9/29/2022 7:30 PM, Sharma, Shashank wrote:



On 9/29/2022 3:37 PM, Lazar, Lijo wrote:

To be clear your understanding -

Nothing is automatic in PMFW. PMFW picks a priority based on the
actual mask sent by driver.

Assuming lower bits corresponds to highest priority -

If driver sends a mask with Bit3 and Bit 0 set, PMFW will chose
profile that corresponds to Bit0. If driver sends a mask with Bit4
Bit2 set and rest unset, PMFW will chose profile that 
corresponds to
Bit2. However if driver sends a mask only with a single bit 
set, it
chooses the profile regardless of whatever was the previous 
profile. t
doesn't check if the existing profile > newly requested one. 
That is

the behavior.

So if a job send chooses a profile that corresponds to Bit0, 
driver

will send that. Next time if another job chooses a profile that
corresponds to Bit1, PMFW will receive that as the new profile and
switch to that. It trusts the driver to send the proper 
workload mask.


Hope that gives the picture.




Thanks, my understanding is also similar, referring to the core 
power

switch profile function here:
amd_powerplay.c::pp_dpm_switch_power_profile()
*snip code*
hwmgr->workload_mask |= (1 << hwmgr->workload_prority[type]);
  index = fls(hwmgr->workload_mask);
  index = index <= Workload_Policy_Max ? index - 1 : 0;
  workload = hwmgr->workload_setting[index];
*snip_code*
hwmgr->hwmgr_func->set_power_profile_mode(hwmgr, , 0);

Here I can see that the new workload mask is appended into the 
existing

workload mask (not overwritten). So if we keep sending new
workload_modes, they would be appended into the workload flags and
finally the PM will pick the most aggressive one of all these 
flags, as

per its policy.



Actually it's misleading -

The path for sienna is -
set_power_profile_mode -> sienna_cichlid_set_power_profile_mode


This code here is a picking one based on lookup table.

   workload_type = smu_cmn_to_asic_specific_index(smu,

CMN2ASIC_MAPPING_WORKLOAD,

smu->power_profile_mode);

This is that lookup table -

static struct cmn2asic_mapping
sienna_cichlid_workload_map[PP_SMC_POWER_PROFILE_COUNT] = {
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT,
WORKLOAD_PPLIB_DEFAULT_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_FULLSCREEN3D,
WORKLOAD_PPLIB_FULL_SCREEN_3D_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_POWERSAVING,
WORKLOAD_PPLIB_POWER_SAVING_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_VIDEO,
WORKLOAD_PPLIB_VIDEO_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_VR,
WORKLOAD_PPLIB_VR_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_COMPUTE,
WORKLOAD_PPLIB_COMPUTE_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_CUSTOM,
WORKLOAD_PPLIB_CUSTOM_BIT),
};


And this is the place of interaction with PMFW. (1 << 
workload_type) is

the mask being sent.

 smu_cmn_send_smc_msg_with_param(smu, 
SMU_MSG_SetWorkloadMask,

  1 << workload_type, NULL);

In the end, driver implementation expects only one bit to be set.


Shashank and I had a discussion about this today.  I think there 
are a

few thing we can do to handle this better:

1. Set a flag that if the user changes the default via sysfs that
overrides any runtime setting via an application since presumably 
that

is what the user wants and we won't change the hint at runtime.
2. Drop the GET API.  There's no need for this, the hint is just a 
hint.


Double checked again based on Felix's comments on API definition. 
Driver decides the priority instead of FW. That way we can still 
keep Get API.



2. Have the driver arbitrate between the available workload profiles
based on the numeric value of the hint (e.g., default < 3D < video <
VR < compute) as the higher values are more aggressive in most cases.
If requests come in for 3D and compute at the same time, the driver
will select compute because it's value is highest.  Each hint type
would be ref counted so we'd know what state to be in every time 
we go

to set the state.  If all of the clients requesting compute go away,
and only 3D requestors remain, we can switch to 3D.  If all refcounts
go to 0, we go back to default.  This will not require any change to
the current workload API in the SMU code.


Since PM layer decides priority, refcount can be kept at powerplay 
and swsmu layer instead of any higher level API.


User API may keep something like req_power_profile (for any 
logging/debug purpose) for the job preference.


No, I think there has been enough confusion around this 
implementation so far, we will implement t

Re: [PATCH v3 5/5] drm/amdgpu: switch workload context to/from compute

[Sharma, Shashank]

On 9/30/2022 11:13 AM, Lazar, Lijo wrote:



On 9/30/2022 2:07 PM, Sharma, Shashank wrote:



On 9/30/2022 7:08 AM, Lazar, Lijo wrote:



On 9/30/2022 12:02 AM, Alex Deucher wrote:
On Thu, Sep 29, 2022 at 10:14 AM Lazar, Lijo  
wrote:




On 9/29/2022 7:30 PM, Sharma, Shashank wrote:



On 9/29/2022 3:37 PM, Lazar, Lijo wrote:

To be clear your understanding -

Nothing is automatic in PMFW. PMFW picks a priority based on the
actual mask sent by driver.

Assuming lower bits corresponds to highest priority -

If driver sends a mask with Bit3 and Bit 0 set, PMFW will chose
profile that corresponds to Bit0. If driver sends a mask with Bit4
Bit2 set and rest unset, PMFW will chose profile that corresponds to
Bit2. However if driver sends a mask only with a single bit set, it
chooses the profile regardless of whatever was the previous 
profile. t

doesn't check if the existing profile > newly requested one. That is
the behavior.

So if a job send chooses a profile that corresponds to Bit0, driver
will send that. Next time if another job chooses a profile that
corresponds to Bit1, PMFW will receive that as the new profile and
switch to that. It trusts the driver to send the proper workload 
mask.


Hope that gives the picture.




Thanks, my understanding is also similar, referring to the core power
switch profile function here:
amd_powerplay.c::pp_dpm_switch_power_profile()
*snip code*
hwmgr->workload_mask |= (1 << hwmgr->workload_prority[type]);
  index = fls(hwmgr->workload_mask);
  index = index <= Workload_Policy_Max ? index - 1 : 0;
  workload = hwmgr->workload_setting[index];
*snip_code*
hwmgr->hwmgr_func->set_power_profile_mode(hwmgr, , 0);

Here I can see that the new workload mask is appended into the 
existing

workload mask (not overwritten). So if we keep sending new
workload_modes, they would be appended into the workload flags and
finally the PM will pick the most aggressive one of all these 
flags, as

per its policy.



Actually it's misleading -

The path for sienna is -
set_power_profile_mode -> sienna_cichlid_set_power_profile_mode


This code here is a picking one based on lookup table.

   workload_type = smu_cmn_to_asic_specific_index(smu,

CMN2ASIC_MAPPING_WORKLOAD,

smu->power_profile_mode);

This is that lookup table -

static struct cmn2asic_mapping
sienna_cichlid_workload_map[PP_SMC_POWER_PROFILE_COUNT] = {
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT,
WORKLOAD_PPLIB_DEFAULT_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_FULLSCREEN3D,
WORKLOAD_PPLIB_FULL_SCREEN_3D_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_POWERSAVING,
WORKLOAD_PPLIB_POWER_SAVING_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_VIDEO,
WORKLOAD_PPLIB_VIDEO_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_VR,
WORKLOAD_PPLIB_VR_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_COMPUTE,
WORKLOAD_PPLIB_COMPUTE_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_CUSTOM,
WORKLOAD_PPLIB_CUSTOM_BIT),
};


And this is the place of interaction with PMFW. (1 << 
workload_type) is

the mask being sent.

 smu_cmn_send_smc_msg_with_param(smu, SMU_MSG_SetWorkloadMask,
  1 << workload_type, NULL);

In the end, driver implementation expects only one bit to be set.


Shashank and I had a discussion about this today.  I think there are a
few thing we can do to handle this better:

1. Set a flag that if the user changes the default via sysfs that
overrides any runtime setting via an application since presumably that
is what the user wants and we won't change the hint at runtime.
2. Drop the GET API.  There's no need for this, the hint is just a 
hint.


Double checked again based on Felix's comments on API definition. 
Driver decides the priority instead of FW. That way we can still keep 
Get API.



2. Have the driver arbitrate between the available workload profiles
based on the numeric value of the hint (e.g., default < 3D < video <
VR < compute) as the higher values are more aggressive in most cases.
If requests come in for 3D and compute at the same time, the driver
will select compute because it's value is highest.  Each hint type
would be ref counted so we'd know what state to be in every time we go
to set the state.  If all of the clients requesting compute go away,
and only 3D requestors remain, we can switch to 3D.  If all refcounts
go to 0, we go back to default.  This will not require any change to
the current workload API in the SMU code.


Since PM layer decides priority, refcount can be kept at powerplay 
and swsmu layer instead of any higher level API.


User API may keep something like req_power_profile (for any 
logging/debug purpose) for the job preference.


No, I think there has been enough confusion around this implementation 
so far, we will implement this just as Alex/Felix suggested:

- No change will be done in pm/SMU layer.


Well, a confusion doesn't j

Re: [PATCH v3 5/5] drm/amdgpu: switch workload context to/from compute

[Sharma, Shashank]

On 9/30/2022 7:08 AM, Lazar, Lijo wrote:



On 9/30/2022 12:02 AM, Alex Deucher wrote:

On Thu, Sep 29, 2022 at 10:14 AM Lazar, Lijo  wrote:




On 9/29/2022 7:30 PM, Sharma, Shashank wrote:



On 9/29/2022 3:37 PM, Lazar, Lijo wrote:

To be clear your understanding -

Nothing is automatic in PMFW. PMFW picks a priority based on the
actual mask sent by driver.

Assuming lower bits corresponds to highest priority -

If driver sends a mask with Bit3 and Bit 0 set, PMFW will chose
profile that corresponds to Bit0. If driver sends a mask with Bit4
Bit2 set and rest unset, PMFW will chose profile that corresponds to
Bit2. However if driver sends a mask only with a single bit set, it
chooses the profile regardless of whatever was the previous profile. t
doesn't check if the existing profile > newly requested one. That is
the behavior.

So if a job send chooses a profile that corresponds to Bit0, driver
will send that. Next time if another job chooses a profile that
corresponds to Bit1, PMFW will receive that as the new profile and
switch to that. It trusts the driver to send the proper workload mask.

Hope that gives the picture.




Thanks, my understanding is also similar, referring to the core power
switch profile function here:
amd_powerplay.c::pp_dpm_switch_power_profile()
*snip code*
hwmgr->workload_mask |= (1 << hwmgr->workload_prority[type]);
  index = fls(hwmgr->workload_mask);
  index = index <= Workload_Policy_Max ? index - 1 : 0;
  workload = hwmgr->workload_setting[index];
*snip_code*
hwmgr->hwmgr_func->set_power_profile_mode(hwmgr, , 0);

Here I can see that the new workload mask is appended into the existing
workload mask (not overwritten). So if we keep sending new
workload_modes, they would be appended into the workload flags and
finally the PM will pick the most aggressive one of all these flags, as
per its policy.



Actually it's misleading -

The path for sienna is -
set_power_profile_mode -> sienna_cichlid_set_power_profile_mode


This code here is a picking one based on lookup table.

   workload_type = smu_cmn_to_asic_specific_index(smu,

CMN2ASIC_MAPPING_WORKLOAD,

smu->power_profile_mode);

This is that lookup table -

static struct cmn2asic_mapping
sienna_cichlid_workload_map[PP_SMC_POWER_PROFILE_COUNT] = {
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT,
WORKLOAD_PPLIB_DEFAULT_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_FULLSCREEN3D,
WORKLOAD_PPLIB_FULL_SCREEN_3D_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_POWERSAVING,
WORKLOAD_PPLIB_POWER_SAVING_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_VIDEO,
WORKLOAD_PPLIB_VIDEO_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_VR,
WORKLOAD_PPLIB_VR_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_COMPUTE,
WORKLOAD_PPLIB_COMPUTE_BIT),
  WORKLOAD_MAP(PP_SMC_POWER_PROFILE_CUSTOM,
WORKLOAD_PPLIB_CUSTOM_BIT),
};


And this is the place of interaction with PMFW. (1 << workload_type) is
the mask being sent.

 smu_cmn_send_smc_msg_with_param(smu, SMU_MSG_SetWorkloadMask,
  1 << workload_type, NULL);

In the end, driver implementation expects only one bit to be set.


Shashank and I had a discussion about this today.  I think there are a
few thing we can do to handle this better:

1. Set a flag that if the user changes the default via sysfs that
overrides any runtime setting via an application since presumably that
is what the user wants and we won't change the hint at runtime.
2. Drop the GET API.  There's no need for this, the hint is just a hint.


Double checked again based on Felix's comments on API definition. Driver 
decides the priority instead of FW. That way we can still keep Get API.



2. Have the driver arbitrate between the available workload profiles
based on the numeric value of the hint (e.g., default < 3D < video <
VR < compute) as the higher values are more aggressive in most cases.
If requests come in for 3D and compute at the same time, the driver
will select compute because it's value is highest.  Each hint type
would be ref counted so we'd know what state to be in every time we go
to set the state.  If all of the clients requesting compute go away,
and only 3D requestors remain, we can switch to 3D.  If all refcounts
go to 0, we go back to default.  This will not require any change to
the current workload API in the SMU code.


Since PM layer decides priority, refcount can be kept at powerplay and 
swsmu layer instead of any higher level API.


User API may keep something like req_power_profile (for any 
logging/debug purpose) for the job preference.


No, I think there has been enough confusion around this implementation 
so far, we will implement this just as Alex/Felix suggested:

- No change will be done in pm/SMU layer.
- The amdgpu_context_workload layer will keep the ref_counting and 
user_workload_hint management, and it will just call and consume t

Re: [PATCH v3 5/5] drm/amdgpu: switch workload context to/from compute

[Sharma, Shashank]

On 9/29/2022 4:14 PM, Lazar, Lijo wrote:



On 9/29/2022 7:30 PM, Sharma, Shashank wrote:



On 9/29/2022 3:37 PM, Lazar, Lijo wrote:

To be clear your understanding -

Nothing is automatic in PMFW. PMFW picks a priority based on the 
actual mask sent by driver.


Assuming lower bits corresponds to highest priority -

If driver sends a mask with Bit3 and Bit 0 set, PMFW will chose 
profile that corresponds to Bit0. If driver sends a mask with Bit4 
Bit2 set and rest unset, PMFW will chose profile that corresponds to 
Bit2. However if driver sends a mask only with a single bit set, it 
chooses the profile regardless of whatever was the previous profile. 
t doesn't check if the existing profile > newly requested one. That 
is the behavior.


So if a job send chooses a profile that corresponds to Bit0, driver 
will send that. Next time if another job chooses a profile that 
corresponds to Bit1, PMFW will receive that as the new profile and 
switch to that. It trusts the driver to send the proper workload mask.


Hope that gives the picture.




Thanks, my understanding is also similar, referring to the core power 
switch profile function here: 
amd_powerplay.c::pp_dpm_switch_power_profile()

*snip code*
hwmgr->workload_mask |= (1 << hwmgr->workload_prority[type]);
 index = fls(hwmgr->workload_mask);
 index = index <= Workload_Policy_Max ? index - 1 : 0;
 workload = hwmgr->workload_setting[index];
*snip_code*
hwmgr->hwmgr_func->set_power_profile_mode(hwmgr, , 0);

Here I can see that the new workload mask is appended into the 
existing workload mask (not overwritten). So if we keep sending new 
workload_modes, they would be appended into the workload flags and 
finally the PM will pick the most aggressive one of all these flags, 
as per its policy.




Actually it's misleading -

The path for sienna is -
set_power_profile_mode -> sienna_cichlid_set_power_profile_mode


This code here is a picking one based on lookup table.

  workload_type = smu_cmn_to_asic_specific_index(smu,

CMN2ASIC_MAPPING_WORKLOAD,

smu->power_profile_mode);

This is that lookup table -

static struct cmn2asic_mapping 
sienna_cichlid_workload_map[PP_SMC_POWER_PROFILE_COUNT] = {
     WORKLOAD_MAP(PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT, 
WORKLOAD_PPLIB_DEFAULT_BIT),
     WORKLOAD_MAP(PP_SMC_POWER_PROFILE_FULLSCREEN3D, 
WORKLOAD_PPLIB_FULL_SCREEN_3D_BIT),
     WORKLOAD_MAP(PP_SMC_POWER_PROFILE_POWERSAVING, 
WORKLOAD_PPLIB_POWER_SAVING_BIT),
     WORKLOAD_MAP(PP_SMC_POWER_PROFILE_VIDEO, 
WORKLOAD_PPLIB_VIDEO_BIT),

     WORKLOAD_MAP(PP_SMC_POWER_PROFILE_VR, WORKLOAD_PPLIB_VR_BIT),
     WORKLOAD_MAP(PP_SMC_POWER_PROFILE_COMPUTE, 
WORKLOAD_PPLIB_COMPUTE_BIT),
     WORKLOAD_MAP(PP_SMC_POWER_PROFILE_CUSTOM, 
WORKLOAD_PPLIB_CUSTOM_BIT),

};


And this is the place of interaction with PMFW. (1 << workload_type) is 
the mask being sent.


    smu_cmn_send_smc_msg_with_param(smu, SMU_MSG_SetWorkloadMask,
     1 << workload_type, NULL);

In the end, driver implementation expects only one bit to be set.



Well this seems like a bug here in the core functions, because the 
powerplay layer is doing the right thing by appending the workload flags 
keeping in mind that a profile_change can be requested while one profile 
is active, but the core functions are actually ignoring those flags.


This brings us to look into actual PM FW expectations. If it expects 
only one flag to be set in the power_mode change message, we don't need 
to bother about this anymore. But if it can handle more than one flag 
but the core driver implementation is blocking it, we will have to fix 
that as well.


@Alex: How can we get more information on this ?

- Shashank


Thanks,
Lijo


Now, when we have a single workload:
-> Job1: requests profile P1 via UAPI, ref count = 1
-> driver sends flags for p1
-> PM FW applies profile P1
-> Job executes in profile P1
-> Job goes to reset function, ref_count = 0,
-> Power profile resets

Now, we have conflicts only when we see multiple workloads (Job1 and 
Job 2)

-> Job1: requests profile P1 via UAPI, ref count = 1
-> driver sends flags for p1
-> PM FW applies profile P1
-> Job executes in profile P1
-> Job2: requests profile P2 via UAPI, refcount = 2
-> driver sends flags for (P1|P2)
-> PM FW picks the more aggressive of the two (Say P1, stays in P1)
-> Job1 goes to reset function, ref_count = 1, job1 does not reset 
power profile
-> Job2 goes to reset function, ref_counter = 2, job 2 resets Power 
profile

-> Power profile resets to None

So this state machine looks like if there is only 1 job, it will be 
executed in desired mode. But if there are multiple, the most 
aggressive profile will be picked, and every job will be executed in 
atleast the requested power profile mode or higher.


Do you find any problem so far ?

- Shashank



Thanks,
Lijo

Re: [PATCH v3 5/5] drm/amdgpu: switch workload context to/from compute

[Sharma, Shashank]

On 9/29/2022 3:37 PM, Lazar, Lijo wrote:

To be clear your understanding -

Nothing is automatic in PMFW. PMFW picks a priority based on the actual 
mask sent by driver.


Assuming lower bits corresponds to highest priority -

If driver sends a mask with Bit3 and Bit 0 set, PMFW will chose profile 
that corresponds to Bit0. If driver sends a mask with Bit4 Bit2 set and 
rest unset, PMFW will chose profile that corresponds to Bit2. However if 
driver sends a mask only with a single bit set, it chooses the profile 
regardless of whatever was the previous profile. t doesn't check if the 
existing profile > newly requested one. That is the behavior.


So if a job send chooses a profile that corresponds to Bit0, driver will 
send that. Next time if another job chooses a profile that corresponds 
to Bit1, PMFW will receive that as the new profile and switch to that. 
It trusts the driver to send the proper workload mask.


Hope that gives the picture.




Thanks, my understanding is also similar, referring to the core power 
switch profile function here: amd_powerplay.c::pp_dpm_switch_power_profile()

*snip code*
hwmgr->workload_mask |= (1 << hwmgr->workload_prority[type]);
index = fls(hwmgr->workload_mask);
index = index <= Workload_Policy_Max ? index - 1 : 0;
workload = hwmgr->workload_setting[index];
*snip_code*
hwmgr->hwmgr_func->set_power_profile_mode(hwmgr, , 0);

Here I can see that the new workload mask is appended into the existing 
workload mask (not overwritten). So if we keep sending new 
workload_modes, they would be appended into the workload flags and 
finally the PM will pick the most aggressive one of all these flags, as 
per its policy.


Now, when we have a single workload:
-> Job1: requests profile P1 via UAPI, ref count = 1
-> driver sends flags for p1
-> PM FW applies profile P1
-> Job executes in profile P1
-> Job goes to reset function, ref_count = 0,
-> Power profile resets

Now, we have conflicts only when we see multiple workloads (Job1 and Job 2)
-> Job1: requests profile P1 via UAPI, ref count = 1
-> driver sends flags for p1
-> PM FW applies profile P1
-> Job executes in profile P1
-> Job2: requests profile P2 via UAPI, refcount = 2
-> driver sends flags for (P1|P2)
-> PM FW picks the more aggressive of the two (Say P1, stays in P1)
-> Job1 goes to reset function, ref_count = 1, job1 does not reset power 
profile

-> Job2 goes to reset function, ref_counter = 2, job 2 resets Power profile
-> Power profile resets to None

So this state machine looks like if there is only 1 job, it will be 
executed in desired mode. But if there are multiple, the most aggressive 
profile will be picked, and every job will be executed in atleast the 
requested power profile mode or higher.


Do you find any problem so far ?

- Shashank



Thanks,
Lijo

Re: [PATCH v3 5/5] drm/amdgpu: switch workload context to/from compute

[Sharma, Shashank]

On 9/29/2022 1:10 PM, Lazar, Lijo wrote:



On 9/29/2022 2:18 PM, Sharma, Shashank wrote:



On 9/28/2022 11:51 PM, Alex Deucher wrote:

On Wed, Sep 28, 2022 at 4:57 AM Sharma, Shashank
 wrote:




On 9/27/2022 10:40 PM, Alex Deucher wrote:

On Tue, Sep 27, 2022 at 11:38 AM Sharma, Shashank
 wrote:




On 9/27/2022 5:23 PM, Felix Kuehling wrote:

Am 2022-09-27 um 10:58 schrieb Sharma, Shashank:

Hello Felix,

Thank for the review comments.

On 9/27/2022 4:48 PM, Felix Kuehling wrote:

Am 2022-09-27 um 02:12 schrieb Christian König:

Am 26.09.22 um 23:40 schrieb Shashank Sharma:

This patch switches the GPU workload mode to/from
compute mode, while submitting compute workload.

Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 


Feel free to add my acked-by, but Felix should probably take a 
look

as well.


This look OK purely from a compute perspective. But I'm concerned
about the interaction of compute with graphics or multiple 
graphics
contexts submitting work concurrently. They would constantly 
override

or disable each other's workload hints.

For example, you have an amdgpu_ctx with
AMDGPU_CTX_WORKLOAD_HINT_COMPUTE (maybe Vulkan compute) and a KFD
process that also wants the compute profile. Those could be 
different
processes belonging to different users. Say, KFD enables the 
compute
profile first. Then the graphics context submits a job. At the 
start

of the job, the compute profile is enabled. That's a no-op because
KFD already enabled the compute profile. When the job finishes, it
disables the compute profile for everyone, including KFD. That's
unexpected.



In this case, it will not disable the compute profile, as the
reference counter will not be zero. The reset_profile() will 
only act

if the reference counter is 0.


OK, I missed the reference counter.




But I would be happy to get any inputs about a policy which can be
more sustainable and gets better outputs, for example:
- should we not allow a profile change, if a PP mode is already
applied and keep it Early bird basis ?

For example: Policy A
- Job A sets the profile to compute
- Job B tries to set profile to 3D, but we do not allow it as 
job A is

not finished it yet.

Or Policy B: Current one
- Job A sets the profile to compute
- Job B tries to set profile to 3D, and we allow it. Job A also 
runs

in PP 3D
- Job B finishes, but does not reset PP as reference count is 
not zero

due to compute
- Job  A finishes, profile reset to NONE


I think this won't work. As I understand it, the
amdgpu_dpm_switch_power_profile enables and disables individual
profiles. Disabling the 3D profile doesn't disable the compute 
profile

at the same time. I think you'll need one refcount per profile.

Regards,
 Felix


Thanks, This is exactly what I was looking for, I think Alex's 
initial

idea was around it, but I was under the assumption that there is only
one HW profile in SMU which keeps on getting overwritten. This can 
solve
our problems, as I can create an array of reference counters, and 
will

disable only the profile whose reference counter goes 0.


It's been a while since I paged any of this code into my head, but I
believe the actual workload message in the SMU is a mask where you can
specify multiple workload types at the same time and the SMU will
arbitrate between them internally.  E.g., the most aggressive one will
be selected out of the ones specified.  I think in the driver we just
set one bit at a time using the current interface.  It might be better
to change the interface and just ref count the hint types and then
when we call the set function look at the ref counts for each hint
type and set the mask as appropriate.

Alex



Hey Alex,
Thanks for your comment, if that is the case, this current patch series
works straight forward, and no changes would be required. Please let me
know if my understanding is correct:

Assumption: Order of aggression: 3D > Media > Compute

- Job 1: Requests mode compute: PP changed to compute, ref count 1
- Job 2: Requests mode media: PP changed to media, ref count 2
- Job 3: requests mode 3D: PP changed to 3D, ref count 3
- Job 1 finishes, downs ref count to 2, doesn't reset the PP as ref 
> 0,

PP still 3D
- Job 3 finishes, downs ref count to 1, doesn't reset the PP as ref 
> 0,

PP still 3D
- Job 2 finishes, downs ref count to 0, PP changed to NONE,

In this way, every job will be operating in the Power profile of 
desired

aggression or higher, and this API guarantees the execution at-least in
the desired power profile.


I'm not entirely sure on the relative levels of aggression, but I
believe the SMU priorities them by index.  E.g.
#define WORKLOAD_PPLIB_DEFAULT_BIT    0
#define WORKLOAD_PPLIB_FULL_SCREEN_3D_BIT 1
#define WORKLOAD_PPLIB_POWER_SAVING_BIT   2
#define WORKLOAD_PPLIB_VIDEO_BIT  3
#define WORKLOAD_PPLIB_VR_BIT 4
#define WORKLOAD_PPLIB_COMPUTE_BIT    5
#define WORKLOAD_PPLIB_CUSTOM_BIT 6

3D < video < VR 

Re: [PATCH v3 5/5] drm/amdgpu: switch workload context to/from compute

[Sharma, Shashank]

On 9/28/2022 11:51 PM, Alex Deucher wrote:

On Wed, Sep 28, 2022 at 4:57 AM Sharma, Shashank
 wrote:




On 9/27/2022 10:40 PM, Alex Deucher wrote:

On Tue, Sep 27, 2022 at 11:38 AM Sharma, Shashank
 wrote:




On 9/27/2022 5:23 PM, Felix Kuehling wrote:

Am 2022-09-27 um 10:58 schrieb Sharma, Shashank:

Hello Felix,

Thank for the review comments.

On 9/27/2022 4:48 PM, Felix Kuehling wrote:

Am 2022-09-27 um 02:12 schrieb Christian König:

Am 26.09.22 um 23:40 schrieb Shashank Sharma:

This patch switches the GPU workload mode to/from
compute mode, while submitting compute workload.

Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 


Feel free to add my acked-by, but Felix should probably take a look
as well.


This look OK purely from a compute perspective. But I'm concerned
about the interaction of compute with graphics or multiple graphics
contexts submitting work concurrently. They would constantly override
or disable each other's workload hints.

For example, you have an amdgpu_ctx with
AMDGPU_CTX_WORKLOAD_HINT_COMPUTE (maybe Vulkan compute) and a KFD
process that also wants the compute profile. Those could be different
processes belonging to different users. Say, KFD enables the compute
profile first. Then the graphics context submits a job. At the start
of the job, the compute profile is enabled. That's a no-op because
KFD already enabled the compute profile. When the job finishes, it
disables the compute profile for everyone, including KFD. That's
unexpected.



In this case, it will not disable the compute profile, as the
reference counter will not be zero. The reset_profile() will only act
if the reference counter is 0.


OK, I missed the reference counter.




But I would be happy to get any inputs about a policy which can be
more sustainable and gets better outputs, for example:
- should we not allow a profile change, if a PP mode is already
applied and keep it Early bird basis ?

For example: Policy A
- Job A sets the profile to compute
- Job B tries to set profile to 3D, but we do not allow it as job A is
not finished it yet.

Or Policy B: Current one
- Job A sets the profile to compute
- Job B tries to set profile to 3D, and we allow it. Job A also runs
in PP 3D
- Job B finishes, but does not reset PP as reference count is not zero
due to compute
- Job  A finishes, profile reset to NONE


I think this won't work. As I understand it, the
amdgpu_dpm_switch_power_profile enables and disables individual
profiles. Disabling the 3D profile doesn't disable the compute profile
at the same time. I think you'll need one refcount per profile.

Regards,
 Felix


Thanks, This is exactly what I was looking for, I think Alex's initial
idea was around it, but I was under the assumption that there is only
one HW profile in SMU which keeps on getting overwritten. This can solve
our problems, as I can create an array of reference counters, and will
disable only the profile whose reference counter goes 0.


It's been a while since I paged any of this code into my head, but I
believe the actual workload message in the SMU is a mask where you can
specify multiple workload types at the same time and the SMU will
arbitrate between them internally.  E.g., the most aggressive one will
be selected out of the ones specified.  I think in the driver we just
set one bit at a time using the current interface.  It might be better
to change the interface and just ref count the hint types and then
when we call the set function look at the ref counts for each hint
type and set the mask as appropriate.

Alex



Hey Alex,
Thanks for your comment, if that is the case, this current patch series
works straight forward, and no changes would be required. Please let me
know if my understanding is correct:

Assumption: Order of aggression: 3D > Media > Compute

- Job 1: Requests mode compute: PP changed to compute, ref count 1
- Job 2: Requests mode media: PP changed to media, ref count 2
- Job 3: requests mode 3D: PP changed to 3D, ref count 3
- Job 1 finishes, downs ref count to 2, doesn't reset the PP as ref > 0,
PP still 3D
- Job 3 finishes, downs ref count to 1, doesn't reset the PP as ref > 0,
PP still 3D
- Job 2 finishes, downs ref count to 0, PP changed to NONE,

In this way, every job will be operating in the Power profile of desired
aggression or higher, and this API guarantees the execution at-least in
the desired power profile.


I'm not entirely sure on the relative levels of aggression, but I
believe the SMU priorities them by index.  E.g.
#define WORKLOAD_PPLIB_DEFAULT_BIT0
#define WORKLOAD_PPLIB_FULL_SCREEN_3D_BIT 1
#define WORKLOAD_PPLIB_POWER_SAVING_BIT   2
#define WORKLOAD_PPLIB_VIDEO_BIT  3
#define WORKLOAD_PPLIB_VR_BIT 4
#define WORKLOAD_PPLIB_COMPUTE_BIT5
#define WORKLOAD_PPLIB_CUSTOM_BIT 6

3D < video < VR < compute < custom

VR and compute are the most aggressive.  Custom takes preference
because it's user customiza

Re: [PATCH v3 5/5] drm/amdgpu: switch workload context to/from compute

[Sharma, Shashank]

Small correction,

On 9/28/2022 10:56 AM, Sharma, Shashank wrote:



On 9/27/2022 10:40 PM, Alex Deucher wrote:

On Tue, Sep 27, 2022 at 11:38 AM Sharma, Shashank
 wrote:




On 9/27/2022 5:23 PM, Felix Kuehling wrote:

Am 2022-09-27 um 10:58 schrieb Sharma, Shashank:

Hello Felix,

Thank for the review comments.

On 9/27/2022 4:48 PM, Felix Kuehling wrote:

Am 2022-09-27 um 02:12 schrieb Christian König:

Am 26.09.22 um 23:40 schrieb Shashank Sharma:

This patch switches the GPU workload mode to/from
compute mode, while submitting compute workload.

Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 


Feel free to add my acked-by, but Felix should probably take a look
as well.


This look OK purely from a compute perspective. But I'm concerned
about the interaction of compute with graphics or multiple graphics
contexts submitting work concurrently. They would constantly override
or disable each other's workload hints.

For example, you have an amdgpu_ctx with
AMDGPU_CTX_WORKLOAD_HINT_COMPUTE (maybe Vulkan compute) and a KFD
process that also wants the compute profile. Those could be different
processes belonging to different users. Say, KFD enables the compute
profile first. Then the graphics context submits a job. At the start
of the job, the compute profile is enabled. That's a no-op because
KFD already enabled the compute profile. When the job finishes, it
disables the compute profile for everyone, including KFD. That's
unexpected.



In this case, it will not disable the compute profile, as the
reference counter will not be zero. The reset_profile() will only act
if the reference counter is 0.


OK, I missed the reference counter.




But I would be happy to get any inputs about a policy which can be
more sustainable and gets better outputs, for example:
- should we not allow a profile change, if a PP mode is already
applied and keep it Early bird basis ?

For example: Policy A
- Job A sets the profile to compute
- Job B tries to set profile to 3D, but we do not allow it as job A is
not finished it yet.

Or Policy B: Current one
- Job A sets the profile to compute
- Job B tries to set profile to 3D, and we allow it. Job A also runs
in PP 3D
- Job B finishes, but does not reset PP as reference count is not zero
due to compute
- Job  A finishes, profile reset to NONE


I think this won't work. As I understand it, the
amdgpu_dpm_switch_power_profile enables and disables individual
profiles. Disabling the 3D profile doesn't disable the compute profile
at the same time. I think you'll need one refcount per profile.

Regards,
    Felix


Thanks, This is exactly what I was looking for, I think Alex's initial
idea was around it, but I was under the assumption that there is only
one HW profile in SMU which keeps on getting overwritten. This can solve
our problems, as I can create an array of reference counters, and will
disable only the profile whose reference counter goes 0.


It's been a while since I paged any of this code into my head, but I
believe the actual workload message in the SMU is a mask where you can
specify multiple workload types at the same time and the SMU will
arbitrate between them internally.  E.g., the most aggressive one will
be selected out of the ones specified.  I think in the driver we just
set one bit at a time using the current interface.  It might be better
to change the interface and just ref count the hint types and then
when we call the set function look at the ref counts for each hint
type and set the mask as appropriate.

Alex



Hey Alex,
Thanks for your comment, if that is the case, this current patch series 
works straight forward, and no changes would be required. 


Only one change required would be to append the new power profile 
request in the existing power profile mask, instead of overwriting it. 
This is where the current state machine pm.workload_mode would be useful.


- Shashank

Please let me

know if my understanding is correct:

Assumption: Order of aggression: 3D > Media > Compute

- Job 1: Requests mode compute: PP changed to compute, ref count 1
- Job 2: Requests mode media: PP changed to media, ref count 2
- Job 3: requests mode 3D: PP changed to 3D, ref count 3
- Job 1 finishes, downs ref count to 2, doesn't reset the PP as ref > 0, 
PP still 3D

- Job 3 finishes, downs ref count to 1, doesn't reset the PP as ref > 0,
PP still 3D
- Job 2 finishes, downs ref count to 0, PP changed to NONE,

In this way, every job will be operating in the Power profile of desired 
aggression or higher, and this API guarantees the execution at-least in 
the desired power profile.


- Shashank





- Shashank







Or anything else ?

REgards
Shashank



Or you have multiple VCN contexts. When context1 finishes a job, it
disables the VIDEO profile. But context2 still has a job on the other
VCN engine and wants the VIDEO profile to still be enabled.

Regards,
    Felix




Christian.


---
   drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 14 +++

Re: [PATCH v3 5/5] drm/amdgpu: switch workload context to/from compute

[Sharma, Shashank]

On 9/27/2022 10:40 PM, Alex Deucher wrote:

On Tue, Sep 27, 2022 at 11:38 AM Sharma, Shashank
 wrote:




On 9/27/2022 5:23 PM, Felix Kuehling wrote:

Am 2022-09-27 um 10:58 schrieb Sharma, Shashank:

Hello Felix,

Thank for the review comments.

On 9/27/2022 4:48 PM, Felix Kuehling wrote:

Am 2022-09-27 um 02:12 schrieb Christian König:

Am 26.09.22 um 23:40 schrieb Shashank Sharma:

This patch switches the GPU workload mode to/from
compute mode, while submitting compute workload.

Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 


Feel free to add my acked-by, but Felix should probably take a look
as well.


This look OK purely from a compute perspective. But I'm concerned
about the interaction of compute with graphics or multiple graphics
contexts submitting work concurrently. They would constantly override
or disable each other's workload hints.

For example, you have an amdgpu_ctx with
AMDGPU_CTX_WORKLOAD_HINT_COMPUTE (maybe Vulkan compute) and a KFD
process that also wants the compute profile. Those could be different
processes belonging to different users. Say, KFD enables the compute
profile first. Then the graphics context submits a job. At the start
of the job, the compute profile is enabled. That's a no-op because
KFD already enabled the compute profile. When the job finishes, it
disables the compute profile for everyone, including KFD. That's
unexpected.



In this case, it will not disable the compute profile, as the
reference counter will not be zero. The reset_profile() will only act
if the reference counter is 0.


OK, I missed the reference counter.




But I would be happy to get any inputs about a policy which can be
more sustainable and gets better outputs, for example:
- should we not allow a profile change, if a PP mode is already
applied and keep it Early bird basis ?

For example: Policy A
- Job A sets the profile to compute
- Job B tries to set profile to 3D, but we do not allow it as job A is
not finished it yet.

Or Policy B: Current one
- Job A sets the profile to compute
- Job B tries to set profile to 3D, and we allow it. Job A also runs
in PP 3D
- Job B finishes, but does not reset PP as reference count is not zero
due to compute
- Job  A finishes, profile reset to NONE


I think this won't work. As I understand it, the
amdgpu_dpm_switch_power_profile enables and disables individual
profiles. Disabling the 3D profile doesn't disable the compute profile
at the same time. I think you'll need one refcount per profile.

Regards,
Felix


Thanks, This is exactly what I was looking for, I think Alex's initial
idea was around it, but I was under the assumption that there is only
one HW profile in SMU which keeps on getting overwritten. This can solve
our problems, as I can create an array of reference counters, and will
disable only the profile whose reference counter goes 0.


It's been a while since I paged any of this code into my head, but I
believe the actual workload message in the SMU is a mask where you can
specify multiple workload types at the same time and the SMU will
arbitrate between them internally.  E.g., the most aggressive one will
be selected out of the ones specified.  I think in the driver we just
set one bit at a time using the current interface.  It might be better
to change the interface and just ref count the hint types and then
when we call the set function look at the ref counts for each hint
type and set the mask as appropriate.

Alex



Hey Alex,
Thanks for your comment, if that is the case, this current patch series 
works straight forward, and no changes would be required. Please let me 
know if my understanding is correct:


Assumption: Order of aggression: 3D > Media > Compute

- Job 1: Requests mode compute: PP changed to compute, ref count 1
- Job 2: Requests mode media: PP changed to media, ref count 2
- Job 3: requests mode 3D: PP changed to 3D, ref count 3
- Job 1 finishes, downs ref count to 2, doesn't reset the PP as ref > 0, 
PP still 3D

- Job 3 finishes, downs ref count to 1, doesn't reset the PP as ref > 0,
PP still 3D
- Job 2 finishes, downs ref count to 0, PP changed to NONE,

In this way, every job will be operating in the Power profile of desired 
aggression or higher, and this API guarantees the execution at-least in 
the desired power profile.


- Shashank





- Shashank







Or anything else ?

REgards
Shashank



Or you have multiple VCN contexts. When context1 finishes a job, it
disables the VIDEO profile. But context2 still has a job on the other
VCN engine and wants the VIDEO profile to still be enabled.

Regards,
Felix




Christian.


---
   drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 14 +++---
   1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
index 5e53a5293935..1caed319a448 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkf

Re: [PATCH v3 0/5] GPU workload hints for better performance

[Sharma, Shashank]

On 9/27/2022 7:13 PM, Michel Dänzer wrote:

On 2022-09-27 18:59, Sharma, Shashank wrote:

Hey Michel,
Thanks for the review coments.

On 9/27/2022 6:24 PM, Michel Dänzer wrote:

On 2022-09-26 23:40, Shashank Sharma wrote:

AMDGPU SOCs supports dynamic workload based power profiles, which can
provide fine-tuned performance for a particular type of workload.
This patch series adds an interface to set/reset these power profiles
based on the workload type hints. A user can set a hint of workload
type being submistted to GPU, and the driver can dynamically switch
the power profiles which is best suited to this kind of workload.

Currently supported workload profiles are:
"None", "3D", "Video", "VR", "Compute"

V2: This version addresses the review comment from Christian about
chaning the design to set workload mode in a more dynamic method
than during the context creation.

V3: Addressed review comment from Christian, Removed the get_workload()
  calls from UAPI, keeping only the set_workload() call.

Shashank Sharma (5):
    drm/amdgpu: add UAPI for workload hints to ctx ioctl
    drm/amdgpu: add new functions to set GPU power profile
    drm/amdgpu: set GPU workload via ctx IOCTL
    drm/amdgpu: switch GPU workload profile
    drm/amdgpu: switch workload context to/from compute


Where are the corresponding Mesa changes?



This series here was to get the feedback on the kernel side design first. As 
you can see from the patch history, we have already changed the design once and 
this is V2. So I thought it would be a good idea to get the feedback on kernel 
UAPI, before starting sending patches to mesa.


In general, it's not possible to review UAPI without the corresponding 
user-space code. I don't think this is an exception.




Sure, good that we already have got the kernel inputs we wanted, now the 
next version will be with corresponding MESA changes.


- Shashank

Re: [PATCH v3 4/5] drm/amdgpu: switch GPU workload profile

[Sharma, Shashank]

On 9/27/2022 6:33 PM, Michel Dänzer wrote:

On 2022-09-27 13:47, Sharma, Shashank wrote:

On 9/27/2022 12:03 PM, Lazar, Lijo wrote:

On 9/27/2022 3:10 AM, Shashank Sharma wrote:

This patch and switches the GPU workload based profile based
on the workload hint information saved in the workload context.
The workload profile is reset to NONE when the job is done.

Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c   |  2 ++
   drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c |  4 
   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c  | 15 +++
   drivers/gpu/drm/amd/amdgpu/amdgpu_job.h  |  3 +++
   4 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
index b7bae833c804..de906a42144f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
@@ -237,6 +237,8 @@ static int amdgpu_cs_parser_init(struct amdgpu_cs_parser 
*p, union drm_amdgpu_cs
   goto free_all_kdata;
   }
+    p->job->workload_mode = p->ctx->workload_mode;
+
   if (p->uf_entry.tv.bo)
   p->job->uf_addr = uf_offset;
   kvfree(chunk_array);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
index a11cf29bc388..625114804121 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
@@ -55,15 +55,11 @@ int amdgpu_set_workload_profile(struct amdgpu_device *adev,
   mutex_lock(>pm.smu_workload_lock);
-    if (adev->pm.workload_mode == hint)
-    goto unlock;
-


What is the expectation when a GFX job + VCN job together (or in general two 
jobs running in separate schedulers) and each prefers a different workload 
type? FW will switch as requested.


Well, I guess the last switched mode will take over. Do note that like most of 
the PM features, the real benefit of power profiles can be seen with consistant 
and similar workloads running for some time (Like gaming, video playback etc).


Not sure how that's supposed to work on a general purpose system, where there 
are always expected to be multiple processes (one of which being the display 
server) using the GPU for different workloads.

Even in special cases there may be multiple different kinds of workloads 
constantly being used at the same time, e.g. a fullscreen game with live 
streaming / recording using VCN.

It looks like we can accommodate that now, see the recent discussion 
with Felix in the patch 5, where we see that 
"amdgpu_dpm_switch_power_profile enables and disables individual 
profiles,  Disabling the 3D profile doesn't disable the compute profile 
at the same time"


So I think we won't be overwriting but would be enabling/disabling 
individual profiles for compute/3D/MM etc. Of course I will have to 
update the patch series accordingly.




Have you guys considered letting the display server (DRM master) choose the 
profile instead?

This seems to be very good input, in case of a further conflict in 
decision making, we might


as well add this intelligence in DRM master. Would you mind explaining 
this a bit more on how do you think it should be done ?


- Shashank

Re: [PATCH v3 0/5] GPU workload hints for better performance

[Sharma, Shashank]

Hey Michel,
Thanks for the review coments.

On 9/27/2022 6:24 PM, Michel Dänzer wrote:

On 2022-09-26 23:40, Shashank Sharma wrote:

AMDGPU SOCs supports dynamic workload based power profiles, which can
provide fine-tuned performance for a particular type of workload.
This patch series adds an interface to set/reset these power profiles
based on the workload type hints. A user can set a hint of workload
type being submistted to GPU, and the driver can dynamically switch
the power profiles which is best suited to this kind of workload.

Currently supported workload profiles are:
"None", "3D", "Video", "VR", "Compute"

V2: This version addresses the review comment from Christian about
chaning the design to set workload mode in a more dynamic method
than during the context creation.

V3: Addressed review comment from Christian, Removed the get_workload()
 calls from UAPI, keeping only the set_workload() call.

Shashank Sharma (5):
   drm/amdgpu: add UAPI for workload hints to ctx ioctl
   drm/amdgpu: add new functions to set GPU power profile
   drm/amdgpu: set GPU workload via ctx IOCTL
   drm/amdgpu: switch GPU workload profile
   drm/amdgpu: switch workload context to/from compute


Where are the corresponding Mesa changes?


This series here was to get the feedback on the kernel side design 
first. As you can see from the patch history, we have already changed 
the design once and this is V2. So I thought it would be a good idea to 
get the feedback on kernel UAPI, before starting sending patches to 
mesa. The mesa/libdrm changes are ready and I was using those mixed with 
libdrm/test/amdgpu stuff to validate the series.


Now I will fine tune them to match the feedback here, and send the 
updated series.


- Shashank

Re: [PATCH v3 5/5] drm/amdgpu: switch workload context to/from compute

[Sharma, Shashank]

On 9/27/2022 5:23 PM, Felix Kuehling wrote:

Am 2022-09-27 um 10:58 schrieb Sharma, Shashank:

Hello Felix,

Thank for the review comments.

On 9/27/2022 4:48 PM, Felix Kuehling wrote:

Am 2022-09-27 um 02:12 schrieb Christian König:

Am 26.09.22 um 23:40 schrieb Shashank Sharma:

This patch switches the GPU workload mode to/from
compute mode, while submitting compute workload.

Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 


Feel free to add my acked-by, but Felix should probably take a look 
as well.


This look OK purely from a compute perspective. But I'm concerned 
about the interaction of compute with graphics or multiple graphics 
contexts submitting work concurrently. They would constantly override 
or disable each other's workload hints.


For example, you have an amdgpu_ctx with 
AMDGPU_CTX_WORKLOAD_HINT_COMPUTE (maybe Vulkan compute) and a KFD 
process that also wants the compute profile. Those could be different 
processes belonging to different users. Say, KFD enables the compute 
profile first. Then the graphics context submits a job. At the start 
of the job, the compute profile is enabled. That's a no-op because 
KFD already enabled the compute profile. When the job finishes, it 
disables the compute profile for everyone, including KFD. That's 
unexpected.




In this case, it will not disable the compute profile, as the 
reference counter will not be zero. The reset_profile() will only act 
if the reference counter is 0.


OK, I missed the reference counter.




But I would be happy to get any inputs about a policy which can be 
more sustainable and gets better outputs, for example:
- should we not allow a profile change, if a PP mode is already 
applied and keep it Early bird basis ?


For example: Policy A
- Job A sets the profile to compute
- Job B tries to set profile to 3D, but we do not allow it as job A is 
not finished it yet.


Or Policy B: Current one
- Job A sets the profile to compute
- Job B tries to set profile to 3D, and we allow it. Job A also runs 
in PP 3D
- Job B finishes, but does not reset PP as reference count is not zero 
due to compute

- Job  A finishes, profile reset to NONE


I think this won't work. As I understand it, the 
amdgpu_dpm_switch_power_profile enables and disables individual 
profiles. Disabling the 3D profile doesn't disable the compute profile 
at the same time. I think you'll need one refcount per profile.


Regards,
   Felix


Thanks, This is exactly what I was looking for, I think Alex's initial 
idea was around it, but I was under the assumption that there is only 
one HW profile in SMU which keeps on getting overwritten. This can solve 
our problems, as I can create an array of reference counters, and will 
disable only the profile whose reference counter goes 0.


- Shashank







Or anything else ?

REgards
Shashank


Or you have multiple VCN contexts. When context1 finishes a job, it 
disables the VIDEO profile. But context2 still has a job on the other 
VCN engine and wants the VIDEO profile to still be enabled.


Regards,
   Felix




Christian.


---
  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 14 +++---
  1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c

index 5e53a5293935..1caed319a448 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
@@ -34,6 +34,7 @@
  #include "amdgpu_ras.h"
  #include "amdgpu_umc.h"
  #include "amdgpu_reset.h"
+#include "amdgpu_ctx_workload.h"
    /* Total memory size in system memory and all GPU VRAM. Used to
   * estimate worst case amount of memory to reserve for page tables
@@ -703,9 +704,16 @@ int amdgpu_amdkfd_submit_ib(struct 
amdgpu_device *adev,
    void amdgpu_amdkfd_set_compute_idle(struct amdgpu_device *adev, 
bool idle)

  {
-    amdgpu_dpm_switch_power_profile(adev,
-    PP_SMC_POWER_PROFILE_COMPUTE,
-    !idle);
+    int ret;
+
+    if (idle)
+    ret = amdgpu_clear_workload_profile(adev, 
AMDGPU_CTX_WORKLOAD_HINT_COMPUTE);

+    else
+    ret = amdgpu_set_workload_profile(adev, 
AMDGPU_CTX_WORKLOAD_HINT_COMPUTE);

+
+    if (ret)
+    drm_warn(>ddev, "Failed to %s power profile to 
compute mode\n",

+ idle ? "reset" : "set");
  }
    bool amdgpu_amdkfd_is_kfd_vmid(struct amdgpu_device *adev, u32 
vmid)

Re: [PATCH v3 5/5] drm/amdgpu: switch workload context to/from compute

[Sharma, Shashank]

Hello Felix,

Thank for the review comments.

On 9/27/2022 4:48 PM, Felix Kuehling wrote:

Am 2022-09-27 um 02:12 schrieb Christian König:

Am 26.09.22 um 23:40 schrieb Shashank Sharma:

This patch switches the GPU workload mode to/from
compute mode, while submitting compute workload.

Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 


Feel free to add my acked-by, but Felix should probably take a look as 
well.


This look OK purely from a compute perspective. But I'm concerned about 
the interaction of compute with graphics or multiple graphics contexts 
submitting work concurrently. They would constantly override or disable 
each other's workload hints.


For example, you have an amdgpu_ctx with 
AMDGPU_CTX_WORKLOAD_HINT_COMPUTE (maybe Vulkan compute) and a KFD 
process that also wants the compute profile. Those could be different 
processes belonging to different users. Say, KFD enables the compute 
profile first. Then the graphics context submits a job. At the start of 
the job, the compute profile is enabled. That's a no-op because KFD 
already enabled the compute profile. When the job finishes, it disables 
the compute profile for everyone, including KFD. That's unexpected.




In this case, it will not disable the compute profile, as the reference 
counter will not be zero. The reset_profile() will only act if the 
reference counter is 0.


But I would be happy to get any inputs about a policy which can be more 
sustainable and gets better outputs, for example:
- should we not allow a profile change, if a PP mode is already applied 
and keep it Early bird basis ?


For example: Policy A
- Job A sets the profile to compute
- Job B tries to set profile to 3D, but we do not allow it as job A is 
not finished it yet.


Or Policy B: Current one
- Job A sets the profile to compute
- Job B tries to set profile to 3D, and we allow it. Job A also runs in 
PP 3D
- Job B finishes, but does not reset PP as reference count is not zero 
due to compute

- Job  A finishes, profile reset to NONE


Or anything else ?

REgards
Shashank


Or you have multiple VCN contexts. When context1 finishes a job, it 
disables the VIDEO profile. But context2 still has a job on the other 
VCN engine and wants the VIDEO profile to still be enabled.


Regards,
   Felix




Christian.


---
  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 14 +++---
  1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c

index 5e53a5293935..1caed319a448 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
@@ -34,6 +34,7 @@
  #include "amdgpu_ras.h"
  #include "amdgpu_umc.h"
  #include "amdgpu_reset.h"
+#include "amdgpu_ctx_workload.h"
    /* Total memory size in system memory and all GPU VRAM. Used to
   * estimate worst case amount of memory to reserve for page tables
@@ -703,9 +704,16 @@ int amdgpu_amdkfd_submit_ib(struct amdgpu_device 
*adev,
    void amdgpu_amdkfd_set_compute_idle(struct amdgpu_device *adev, 
bool idle)

  {
-    amdgpu_dpm_switch_power_profile(adev,
-    PP_SMC_POWER_PROFILE_COMPUTE,
-    !idle);
+    int ret;
+
+    if (idle)
+    ret = amdgpu_clear_workload_profile(adev, 
AMDGPU_CTX_WORKLOAD_HINT_COMPUTE);

+    else
+    ret = amdgpu_set_workload_profile(adev, 
AMDGPU_CTX_WORKLOAD_HINT_COMPUTE);

+
+    if (ret)
+    drm_warn(>ddev, "Failed to %s power profile to compute 
mode\n",

+ idle ? "reset" : "set");
  }
    bool amdgpu_amdkfd_is_kfd_vmid(struct amdgpu_device *adev, u32 vmid)

Re: [PATCH v3 2/5] drm/amdgpu: add new functions to set GPU power profile

[Sharma, Shashank]

On 9/27/2022 4:34 PM, Lazar, Lijo wrote:



On 9/27/2022 7:50 PM, Sharma, Shashank wrote:



On 9/27/2022 4:00 PM, Lazar, Lijo wrote:



On 9/27/2022 7:17 PM, Sharma, Shashank wrote:



On 9/27/2022 3:29 PM, Lazar, Lijo wrote:



On 9/27/2022 6:23 PM, Sharma, Shashank wrote:



On 9/27/2022 2:39 PM, Lazar, Lijo wrote:



On 9/27/2022 5:53 PM, Sharma, Shashank wrote:



On 9/27/2022 2:10 PM, Lazar, Lijo wrote:



On 9/27/2022 5:11 PM, Sharma, Shashank wrote:



On 9/27/2022 11:58 AM, Lazar, Lijo wrote:



On 9/27/2022 3:10 AM, Shashank Sharma wrote:

This patch adds new functions which will allow a user to
change the GPU power profile based a GPU workload hint
flag.

Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |  2 +-
  .../gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c  | 97 
+++

  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    |  1 +
  .../gpu/drm/amd/include/amdgpu_ctx_workload.h | 54 
+++

  drivers/gpu/drm/amd/pm/inc/amdgpu_dpm.h   |  5 +
  5 files changed, 158 insertions(+), 1 deletion(-)
  create mode 100644 
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
  create mode 100644 
drivers/gpu/drm/amd/include/amdgpu_ctx_workload.h


diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index 5a283d12f8e1..34679c657ecc 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -50,7 +50,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
  atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
  atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
  amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o 
amdgpu_ib.o amdgpu_pll.o \
-    amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o 
amdgpu_sync.o \
+    amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o 
amdgpu_ctx_workload.o amdgpu_sync.o \
  amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o 
amdgpu_vram_mgr.o amdgpu_virt.o \

  amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
  amdgpu_debugfs.o amdgpu_ids.o amdgpu_gmc.o \
diff --git 
a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c

new file mode 100644
index ..a11cf29bc388
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
@@ -0,0 +1,97 @@
+/*
+ * Copyright 2022 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any 
person obtaining a
+ * copy of this software and associated documentation files 
(the "Software"),
+ * to deal in the Software without restriction, including 
without limitation
+ * the rights to use, copy, modify, merge, publish, 
distribute, sublicense,
+ * and/or sell copies of the Software, and to permit 
persons to whom the
+ * Software is furnished to do so, subject to the following 
conditions:

+ *
+ * The above copyright notice and this permission notice 
shall be included in

+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF 
ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 
MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  
IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY 
CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT 
OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE 
OR THE USE OR

+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#include 
+#include "kgd_pp_interface.h"
+#include "amdgpu_ctx_workload.h"
+
+static enum PP_SMC_POWER_PROFILE
+amdgpu_workload_to_power_profile(uint32_t hint)
+{
+    switch (hint) {
+    case AMDGPU_CTX_WORKLOAD_HINT_NONE:
+    default:
+    return PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT;
+
+    case AMDGPU_CTX_WORKLOAD_HINT_3D:
+    return PP_SMC_POWER_PROFILE_FULLSCREEN3D;
+    case AMDGPU_CTX_WORKLOAD_HINT_VIDEO:
+    return PP_SMC_POWER_PROFILE_VIDEO;
+    case AMDGPU_CTX_WORKLOAD_HINT_VR:
+    return PP_SMC_POWER_PROFILE_VR;
+    case AMDGPU_CTX_WORKLOAD_HINT_COMPUTE:
+    return PP_SMC_POWER_PROFILE_COMPUTE;
+    }
+}
+
+int amdgpu_set_workload_profile(struct amdgpu_device *adev,
+    uint32_t hint)
+{
+    int ret = 0;
+    enum PP_SMC_POWER_PROFILE profile =
+    amdgpu_workload_to_power_profile(hint);
+
+    if (adev->pm.workload_mode == hint)
+    return 0;
+
+    mutex_lock(>pm.smu_workload_lock);


If it's all about pm subsystem variable accesses, this API 
should rather be inside amd/pm subsystem. No need to expose 
the variable outside pm subsytem. Also currently all 
amdgpu_dpm* calls are protected under one mutex. Then this 
extra lock won't be needed.




This is tricky, this is not all about PM subsystem. Note that 
the job management and scheduling is handled into amdgpu_ctx, 
so the workload hint is set in context_management API. The API 
is consumed when the job is actuall

Re: [PATCH v3 2/5] drm/amdgpu: add new functions to set GPU power profile

[Sharma, Shashank]

On 9/27/2022 4:00 PM, Lazar, Lijo wrote:



On 9/27/2022 7:17 PM, Sharma, Shashank wrote:



On 9/27/2022 3:29 PM, Lazar, Lijo wrote:



On 9/27/2022 6:23 PM, Sharma, Shashank wrote:



On 9/27/2022 2:39 PM, Lazar, Lijo wrote:



On 9/27/2022 5:53 PM, Sharma, Shashank wrote:



On 9/27/2022 2:10 PM, Lazar, Lijo wrote:



On 9/27/2022 5:11 PM, Sharma, Shashank wrote:



On 9/27/2022 11:58 AM, Lazar, Lijo wrote:



On 9/27/2022 3:10 AM, Shashank Sharma wrote:

This patch adds new functions which will allow a user to
change the GPU power profile based a GPU workload hint
flag.

Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |  2 +-
  .../gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c  | 97 
+++

  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    |  1 +
  .../gpu/drm/amd/include/amdgpu_ctx_workload.h | 54 +++
  drivers/gpu/drm/amd/pm/inc/amdgpu_dpm.h   |  5 +
  5 files changed, 158 insertions(+), 1 deletion(-)
  create mode 100644 
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
  create mode 100644 
drivers/gpu/drm/amd/include/amdgpu_ctx_workload.h


diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index 5a283d12f8e1..34679c657ecc 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -50,7 +50,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
  atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
  atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
  amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o 
amdgpu_pll.o \

-    amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
+    amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o 
amdgpu_ctx_workload.o amdgpu_sync.o \
  amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o 
amdgpu_virt.o \

  amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
  amdgpu_debugfs.o amdgpu_ids.o amdgpu_gmc.o \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c

new file mode 100644
index ..a11cf29bc388
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
@@ -0,0 +1,97 @@
+/*
+ * Copyright 2022 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any 
person obtaining a
+ * copy of this software and associated documentation files 
(the "Software"),
+ * to deal in the Software without restriction, including 
without limitation
+ * the rights to use, copy, modify, merge, publish, 
distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons 
to whom the
+ * Software is furnished to do so, subject to the following 
conditions:

+ *
+ * The above copyright notice and this permission notice 
shall be included in

+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY 
KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 
MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN 
NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY 
CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 
OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR 
THE USE OR

+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#include 
+#include "kgd_pp_interface.h"
+#include "amdgpu_ctx_workload.h"
+
+static enum PP_SMC_POWER_PROFILE
+amdgpu_workload_to_power_profile(uint32_t hint)
+{
+    switch (hint) {
+    case AMDGPU_CTX_WORKLOAD_HINT_NONE:
+    default:
+    return PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT;
+
+    case AMDGPU_CTX_WORKLOAD_HINT_3D:
+    return PP_SMC_POWER_PROFILE_FULLSCREEN3D;
+    case AMDGPU_CTX_WORKLOAD_HINT_VIDEO:
+    return PP_SMC_POWER_PROFILE_VIDEO;
+    case AMDGPU_CTX_WORKLOAD_HINT_VR:
+    return PP_SMC_POWER_PROFILE_VR;
+    case AMDGPU_CTX_WORKLOAD_HINT_COMPUTE:
+    return PP_SMC_POWER_PROFILE_COMPUTE;
+    }
+}
+
+int amdgpu_set_workload_profile(struct amdgpu_device *adev,
+    uint32_t hint)
+{
+    int ret = 0;
+    enum PP_SMC_POWER_PROFILE profile =
+    amdgpu_workload_to_power_profile(hint);
+
+    if (adev->pm.workload_mode == hint)
+    return 0;
+
+    mutex_lock(>pm.smu_workload_lock);


If it's all about pm subsystem variable accesses, this API 
should rather be inside amd/pm subsystem. No need to expose the 
variable outside pm subsytem. Also currently all amdgpu_dpm* 
calls are protected under one mutex. Then this extra lock won't 
be needed.




This is tricky, this is not all about PM subsystem. Note that 
the job management and scheduling is handled into amdgpu_ctx, so 
the workload hint is set in context_management API. The API is 
consumed when the job is actually run from amdgpu_run() layer. 
So its a joint interface between context and PM.




If you ta

Re: [PATCH v3 2/5] drm/amdgpu: add new functions to set GPU power profile

[Sharma, Shashank]

On 9/27/2022 3:29 PM, Lazar, Lijo wrote:



On 9/27/2022 6:23 PM, Sharma, Shashank wrote:



On 9/27/2022 2:39 PM, Lazar, Lijo wrote:



On 9/27/2022 5:53 PM, Sharma, Shashank wrote:



On 9/27/2022 2:10 PM, Lazar, Lijo wrote:



On 9/27/2022 5:11 PM, Sharma, Shashank wrote:



On 9/27/2022 11:58 AM, Lazar, Lijo wrote:



On 9/27/2022 3:10 AM, Shashank Sharma wrote:

This patch adds new functions which will allow a user to
change the GPU power profile based a GPU workload hint
flag.

Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |  2 +-
  .../gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c  | 97 
+++

  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    |  1 +
  .../gpu/drm/amd/include/amdgpu_ctx_workload.h | 54 +++
  drivers/gpu/drm/amd/pm/inc/amdgpu_dpm.h   |  5 +
  5 files changed, 158 insertions(+), 1 deletion(-)
  create mode 100644 
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
  create mode 100644 
drivers/gpu/drm/amd/include/amdgpu_ctx_workload.h


diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index 5a283d12f8e1..34679c657ecc 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -50,7 +50,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
  atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
  atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
  amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o 
amdgpu_pll.o \

-    amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
+    amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o 
amdgpu_ctx_workload.o amdgpu_sync.o \
  amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o 
amdgpu_virt.o \

  amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
  amdgpu_debugfs.o amdgpu_ids.o amdgpu_gmc.o \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c

new file mode 100644
index ..a11cf29bc388
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
@@ -0,0 +1,97 @@
+/*
+ * Copyright 2022 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person 
obtaining a
+ * copy of this software and associated documentation files 
(the "Software"),
+ * to deal in the Software without restriction, including 
without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, 
sublicense,
+ * and/or sell copies of the Software, and to permit persons to 
whom the
+ * Software is furnished to do so, subject to the following 
conditions:

+ *
+ * The above copyright notice and this permission notice shall 
be included in

+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY 
KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 
MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO 
EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY 
CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 
OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR 
THE USE OR

+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#include 
+#include "kgd_pp_interface.h"
+#include "amdgpu_ctx_workload.h"
+
+static enum PP_SMC_POWER_PROFILE
+amdgpu_workload_to_power_profile(uint32_t hint)
+{
+    switch (hint) {
+    case AMDGPU_CTX_WORKLOAD_HINT_NONE:
+    default:
+    return PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT;
+
+    case AMDGPU_CTX_WORKLOAD_HINT_3D:
+    return PP_SMC_POWER_PROFILE_FULLSCREEN3D;
+    case AMDGPU_CTX_WORKLOAD_HINT_VIDEO:
+    return PP_SMC_POWER_PROFILE_VIDEO;
+    case AMDGPU_CTX_WORKLOAD_HINT_VR:
+    return PP_SMC_POWER_PROFILE_VR;
+    case AMDGPU_CTX_WORKLOAD_HINT_COMPUTE:
+    return PP_SMC_POWER_PROFILE_COMPUTE;
+    }
+}
+
+int amdgpu_set_workload_profile(struct amdgpu_device *adev,
+    uint32_t hint)
+{
+    int ret = 0;
+    enum PP_SMC_POWER_PROFILE profile =
+    amdgpu_workload_to_power_profile(hint);
+
+    if (adev->pm.workload_mode == hint)
+    return 0;
+
+    mutex_lock(>pm.smu_workload_lock);


If it's all about pm subsystem variable accesses, this API should 
rather be inside amd/pm subsystem. No need to expose the variable 
outside pm subsytem. Also currently all amdgpu_dpm* calls are 
protected under one mutex. Then this extra lock won't be needed.




This is tricky, this is not all about PM subsystem. Note that the 
job management and scheduling is handled into amdgpu_ctx, so the 
workload hint is set in context_management API. The API is 
consumed when the job is actually run from amdgpu_run() layer. So 
its a joint interface between context and PM.




If you take out amdgpu_workload_to_power_profile() line, everything 
else looks to touch only pm variables/

Re: [PATCH v3 2/5] drm/amdgpu: add new functions to set GPU power profile

[Sharma, Shashank]

On 9/27/2022 2:39 PM, Lazar, Lijo wrote:



On 9/27/2022 5:53 PM, Sharma, Shashank wrote:



On 9/27/2022 2:10 PM, Lazar, Lijo wrote:



On 9/27/2022 5:11 PM, Sharma, Shashank wrote:



On 9/27/2022 11:58 AM, Lazar, Lijo wrote:



On 9/27/2022 3:10 AM, Shashank Sharma wrote:

This patch adds new functions which will allow a user to
change the GPU power profile based a GPU workload hint
flag.

Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |  2 +-
  .../gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c  | 97 
+++

  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    |  1 +
  .../gpu/drm/amd/include/amdgpu_ctx_workload.h | 54 +++
  drivers/gpu/drm/amd/pm/inc/amdgpu_dpm.h   |  5 +
  5 files changed, 158 insertions(+), 1 deletion(-)
  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
  create mode 100644 
drivers/gpu/drm/amd/include/amdgpu_ctx_workload.h


diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index 5a283d12f8e1..34679c657ecc 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -50,7 +50,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
  atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
  atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
  amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o 
amdgpu_pll.o \

-    amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
+    amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o 
amdgpu_ctx_workload.o amdgpu_sync.o \
  amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o 
amdgpu_virt.o \

  amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
  amdgpu_debugfs.o amdgpu_ids.o amdgpu_gmc.o \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c

new file mode 100644
index ..a11cf29bc388
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
@@ -0,0 +1,97 @@
+/*
+ * Copyright 2022 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person 
obtaining a
+ * copy of this software and associated documentation files (the 
"Software"),
+ * to deal in the Software without restriction, including without 
limitation
+ * the rights to use, copy, modify, merge, publish, distribute, 
sublicense,
+ * and/or sell copies of the Software, and to permit persons to 
whom the
+ * Software is furnished to do so, subject to the following 
conditions:

+ *
+ * The above copyright notice and this permission notice shall be 
included in

+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY 
KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 
MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO 
EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, 
DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 
OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE 
USE OR

+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#include 
+#include "kgd_pp_interface.h"
+#include "amdgpu_ctx_workload.h"
+
+static enum PP_SMC_POWER_PROFILE
+amdgpu_workload_to_power_profile(uint32_t hint)
+{
+    switch (hint) {
+    case AMDGPU_CTX_WORKLOAD_HINT_NONE:
+    default:
+    return PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT;
+
+    case AMDGPU_CTX_WORKLOAD_HINT_3D:
+    return PP_SMC_POWER_PROFILE_FULLSCREEN3D;
+    case AMDGPU_CTX_WORKLOAD_HINT_VIDEO:
+    return PP_SMC_POWER_PROFILE_VIDEO;
+    case AMDGPU_CTX_WORKLOAD_HINT_VR:
+    return PP_SMC_POWER_PROFILE_VR;
+    case AMDGPU_CTX_WORKLOAD_HINT_COMPUTE:
+    return PP_SMC_POWER_PROFILE_COMPUTE;
+    }
+}
+
+int amdgpu_set_workload_profile(struct amdgpu_device *adev,
+    uint32_t hint)
+{
+    int ret = 0;
+    enum PP_SMC_POWER_PROFILE profile =
+    amdgpu_workload_to_power_profile(hint);
+
+    if (adev->pm.workload_mode == hint)
+    return 0;
+
+    mutex_lock(>pm.smu_workload_lock);


If it's all about pm subsystem variable accesses, this API should 
rather be inside amd/pm subsystem. No need to expose the variable 
outside pm subsytem. Also currently all amdgpu_dpm* calls are 
protected under one mutex. Then this extra lock won't be needed.




This is tricky, this is not all about PM subsystem. Note that the 
job management and scheduling is handled into amdgpu_ctx, so the 
workload hint is set in context_management API. The API is consumed 
when the job is actually run from amdgpu_run() layer. So its a joint 
interface between context and PM.




If you take out amdgpu_workload_to_power_profile() line, everything 
else looks to touch only pm variables/functions. 


That's not a line, that function converts a AMGPU_CTX hint to PPM 
profile. And

Re: [PATCH v3 4/5] drm/amdgpu: switch GPU workload profile

[Sharma, Shashank]

On 9/27/2022 2:20 PM, Lazar, Lijo wrote:



On 9/27/2022 5:17 PM, Sharma, Shashank wrote:



On 9/27/2022 12:03 PM, Lazar, Lijo wrote:



On 9/27/2022 3:10 AM, Shashank Sharma wrote:

This patch and switches the GPU workload based profile based
on the workload hint information saved in the workload context.
The workload profile is reset to NONE when the job is done.

Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c   |  2 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c |  4 
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c  | 15 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.h  |  3 +++
  4 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c

index b7bae833c804..de906a42144f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
@@ -237,6 +237,8 @@ static int amdgpu_cs_parser_init(struct 
amdgpu_cs_parser *p, union drm_amdgpu_cs

  goto free_all_kdata;
  }
+    p->job->workload_mode = p->ctx->workload_mode;
+
  if (p->uf_entry.tv.bo)
  p->job->uf_addr = uf_offset;
  kvfree(chunk_array);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c

index a11cf29bc388..625114804121 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
@@ -55,15 +55,11 @@ int amdgpu_set_workload_profile(struct 
amdgpu_device *adev,

  mutex_lock(>pm.smu_workload_lock);
-    if (adev->pm.workload_mode == hint)
-    goto unlock;
-


What is the expectation when a GFX job + VCN job together (or in 
general two jobs running in separate schedulers) and each prefers a 
different workload type? FW will switch as requested.


Well, I guess the last switched mode will take over. Do note that like 
most of the PM features, the real benefit of power profiles can be 
seen with consistant and similar workloads running for some time (Like 
gaming, video playback etc).




Yes, so the extra protection layer wrapping around this is really not 
helping (user doesn't know if the job is really run in the requested 
mode). I would suggest to avoid that and document the usage of this API 
as exclusive mode usage for some profiling use cases.




As I mentioned in the other comment, this extra protection is not for 
not allowing it to change the mode, but from preventing PM reset from 
job_cleanup thread, while another work is in progress.


- Shashank


Thanks,
Lijo


- Shashank



Thanks,
Lijo


  ret = amdgpu_dpm_switch_power_profile(adev, profile, 1);
  if (!ret)
  adev->pm.workload_mode = hint;
  atomic_inc(>pm.workload_switch_ref);
-unlock:
  mutex_unlock(>pm.smu_workload_lock);
  return ret;
  }
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c

index c2fd6f3076a6..9300e86ee7c5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
@@ -30,6 +30,7 @@
  #include "amdgpu.h"
  #include "amdgpu_trace.h"
  #include "amdgpu_reset.h"
+#include "amdgpu_ctx_workload.h"
  static enum drm_gpu_sched_stat amdgpu_job_timedout(struct 
drm_sched_job *s_job)

  {
@@ -144,6 +145,14 @@ void amdgpu_job_free_resources(struct 
amdgpu_job *job)

  static void amdgpu_job_free_cb(struct drm_sched_job *s_job)
  {
  struct amdgpu_job *job = to_amdgpu_job(s_job);
+    struct amdgpu_ring *ring = to_amdgpu_ring(s_job->sched);
+
+    if (job->workload_mode != AMDGPU_CTX_WORKLOAD_HINT_NONE) {
+    if (amdgpu_clear_workload_profile(ring->adev, 
job->workload_mode))

+    DRM_WARN("Failed to come out of workload profile %s\n",
+    amdgpu_workload_profile_name(job->workload_mode));
+    job->workload_mode = AMDGPU_CTX_WORKLOAD_HINT_NONE;
+    }
  drm_sched_job_cleanup(s_job);
@@ -256,6 +265,12 @@ static struct dma_fence *amdgpu_job_run(struct 
drm_sched_job *sched_job)

  DRM_ERROR("Error scheduling IBs (%d)\n", r);
  }
+    if (job->workload_mode != AMDGPU_CTX_WORKLOAD_HINT_NONE) {
+    if (amdgpu_set_workload_profile(ring->adev, 
job->workload_mode))

+    DRM_WARN("Failed to set workload profile to %s\n",
+  amdgpu_workload_profile_name(job->workload_mode));
+    }
+
  job->job_run_counter++;
  amdgpu_job_free_resources(job);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h

index babc0af751c2..573e8692c814 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
@@ -68,6 +68,9 @@ struct amdgpu_job {
  /* job_run_counter >= 1 means a resubmit job */
 

Re: [PATCH v3 2/5] drm/amdgpu: add new functions to set GPU power profile

[Sharma, Shashank]

On 9/27/2022 2:10 PM, Lazar, Lijo wrote:



On 9/27/2022 5:11 PM, Sharma, Shashank wrote:



On 9/27/2022 11:58 AM, Lazar, Lijo wrote:



On 9/27/2022 3:10 AM, Shashank Sharma wrote:

This patch adds new functions which will allow a user to
change the GPU power profile based a GPU workload hint
flag.

Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |  2 +-
  .../gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c  | 97 
+++

  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    |  1 +
  .../gpu/drm/amd/include/amdgpu_ctx_workload.h | 54 +++
  drivers/gpu/drm/amd/pm/inc/amdgpu_dpm.h   |  5 +
  5 files changed, 158 insertions(+), 1 deletion(-)
  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
  create mode 100644 drivers/gpu/drm/amd/include/amdgpu_ctx_workload.h

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index 5a283d12f8e1..34679c657ecc 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -50,7 +50,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
  atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
  atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
  amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o 
amdgpu_pll.o \

-    amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
+    amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o 
amdgpu_ctx_workload.o amdgpu_sync.o \
  amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o 
amdgpu_virt.o \

  amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
  amdgpu_debugfs.o amdgpu_ids.o amdgpu_gmc.o \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c

new file mode 100644
index ..a11cf29bc388
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
@@ -0,0 +1,97 @@
+/*
+ * Copyright 2022 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person 
obtaining a
+ * copy of this software and associated documentation files (the 
"Software"),
+ * to deal in the Software without restriction, including without 
limitation
+ * the rights to use, copy, modify, merge, publish, distribute, 
sublicense,
+ * and/or sell copies of the Software, and to permit persons to 
whom the
+ * Software is furnished to do so, subject to the following 
conditions:

+ *
+ * The above copyright notice and this permission notice shall be 
included in

+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 
EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 
MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO 
EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, 
DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 
OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE 
USE OR

+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#include 
+#include "kgd_pp_interface.h"
+#include "amdgpu_ctx_workload.h"
+
+static enum PP_SMC_POWER_PROFILE
+amdgpu_workload_to_power_profile(uint32_t hint)
+{
+    switch (hint) {
+    case AMDGPU_CTX_WORKLOAD_HINT_NONE:
+    default:
+    return PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT;
+
+    case AMDGPU_CTX_WORKLOAD_HINT_3D:
+    return PP_SMC_POWER_PROFILE_FULLSCREEN3D;
+    case AMDGPU_CTX_WORKLOAD_HINT_VIDEO:
+    return PP_SMC_POWER_PROFILE_VIDEO;
+    case AMDGPU_CTX_WORKLOAD_HINT_VR:
+    return PP_SMC_POWER_PROFILE_VR;
+    case AMDGPU_CTX_WORKLOAD_HINT_COMPUTE:
+    return PP_SMC_POWER_PROFILE_COMPUTE;
+    }
+}
+
+int amdgpu_set_workload_profile(struct amdgpu_device *adev,
+    uint32_t hint)
+{
+    int ret = 0;
+    enum PP_SMC_POWER_PROFILE profile =
+    amdgpu_workload_to_power_profile(hint);
+
+    if (adev->pm.workload_mode == hint)
+    return 0;
+
+    mutex_lock(>pm.smu_workload_lock);


If it's all about pm subsystem variable accesses, this API should 
rather be inside amd/pm subsystem. No need to expose the variable 
outside pm subsytem. Also currently all amdgpu_dpm* calls are 
protected under one mutex. Then this extra lock won't be needed.




This is tricky, this is not all about PM subsystem. Note that the job 
management and scheduling is handled into amdgpu_ctx, so the workload 
hint is set in context_management API. The API is consumed when the 
job is actually run from amdgpu_run() layer. So its a joint interface 
between context and PM.




If you take out amdgpu_workload_to_power_profile() line, everything else 
looks to touch only pm variables/functions. 


That's not a line, that function converts a AMGPU_CTX hint to PPM 
profile. And going by that logic, this whole code was kept in the 
amdgpu_ctx.c file as well, coz this code is cons

Re: [PATCH v3 4/5] drm/amdgpu: switch GPU workload profile

[Sharma, Shashank]

On 9/27/2022 12:03 PM, Lazar, Lijo wrote:



On 9/27/2022 3:10 AM, Shashank Sharma wrote:

This patch and switches the GPU workload based profile based
on the workload hint information saved in the workload context.
The workload profile is reset to NONE when the job is done.

Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c   |  2 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c |  4 
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c  | 15 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.h  |  3 +++
  4 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c

index b7bae833c804..de906a42144f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
@@ -237,6 +237,8 @@ static int amdgpu_cs_parser_init(struct 
amdgpu_cs_parser *p, union drm_amdgpu_cs

  goto free_all_kdata;
  }
+    p->job->workload_mode = p->ctx->workload_mode;
+
  if (p->uf_entry.tv.bo)
  p->job->uf_addr = uf_offset;
  kvfree(chunk_array);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c

index a11cf29bc388..625114804121 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
@@ -55,15 +55,11 @@ int amdgpu_set_workload_profile(struct 
amdgpu_device *adev,

  mutex_lock(>pm.smu_workload_lock);
-    if (adev->pm.workload_mode == hint)
-    goto unlock;
-


What is the expectation when a GFX job + VCN job together (or in general 
two jobs running in separate schedulers) and each prefers a different 
workload type? FW will switch as requested.


Well, I guess the last switched mode will take over. Do note that like 
most of the PM features, the real benefit of power profiles can be seen 
with consistant and similar workloads running for some time (Like 
gaming, video playback etc).


- Shashank



Thanks,
Lijo


  ret = amdgpu_dpm_switch_power_profile(adev, profile, 1);
  if (!ret)
  adev->pm.workload_mode = hint;
  atomic_inc(>pm.workload_switch_ref);
-unlock:
  mutex_unlock(>pm.smu_workload_lock);
  return ret;
  }
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c

index c2fd6f3076a6..9300e86ee7c5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
@@ -30,6 +30,7 @@
  #include "amdgpu.h"
  #include "amdgpu_trace.h"
  #include "amdgpu_reset.h"
+#include "amdgpu_ctx_workload.h"
  static enum drm_gpu_sched_stat amdgpu_job_timedout(struct 
drm_sched_job *s_job)

  {
@@ -144,6 +145,14 @@ void amdgpu_job_free_resources(struct amdgpu_job 
*job)

  static void amdgpu_job_free_cb(struct drm_sched_job *s_job)
  {
  struct amdgpu_job *job = to_amdgpu_job(s_job);
+    struct amdgpu_ring *ring = to_amdgpu_ring(s_job->sched);
+
+    if (job->workload_mode != AMDGPU_CTX_WORKLOAD_HINT_NONE) {
+    if (amdgpu_clear_workload_profile(ring->adev, 
job->workload_mode))

+    DRM_WARN("Failed to come out of workload profile %s\n",
+    amdgpu_workload_profile_name(job->workload_mode));
+    job->workload_mode = AMDGPU_CTX_WORKLOAD_HINT_NONE;
+    }
  drm_sched_job_cleanup(s_job);
@@ -256,6 +265,12 @@ static struct dma_fence *amdgpu_job_run(struct 
drm_sched_job *sched_job)

  DRM_ERROR("Error scheduling IBs (%d)\n", r);
  }
+    if (job->workload_mode != AMDGPU_CTX_WORKLOAD_HINT_NONE) {
+    if (amdgpu_set_workload_profile(ring->adev, job->workload_mode))
+    DRM_WARN("Failed to set workload profile to %s\n",
+  amdgpu_workload_profile_name(job->workload_mode));
+    }
+
  job->job_run_counter++;
  amdgpu_job_free_resources(job);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h

index babc0af751c2..573e8692c814 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
@@ -68,6 +68,9 @@ struct amdgpu_job {
  /* job_run_counter >= 1 means a resubmit job */
  uint32_t    job_run_counter;
+    /* workload mode hint for pm */
+    uint32_t    workload_mode;
+
  uint32_t    num_ibs;
  struct amdgpu_ib    ibs[];
  };

Re: [PATCH v3 2/5] drm/amdgpu: add new functions to set GPU power profile

[Sharma, Shashank]

On 9/27/2022 11:58 AM, Lazar, Lijo wrote:



On 9/27/2022 3:10 AM, Shashank Sharma wrote:

This patch adds new functions which will allow a user to
change the GPU power profile based a GPU workload hint
flag.

Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |  2 +-
  .../gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c  | 97 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    |  1 +
  .../gpu/drm/amd/include/amdgpu_ctx_workload.h | 54 +++
  drivers/gpu/drm/amd/pm/inc/amdgpu_dpm.h   |  5 +
  5 files changed, 158 insertions(+), 1 deletion(-)
  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
  create mode 100644 drivers/gpu/drm/amd/include/amdgpu_ctx_workload.h

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile 
b/drivers/gpu/drm/amd/amdgpu/Makefile

index 5a283d12f8e1..34679c657ecc 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -50,7 +50,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
  atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
  atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
  amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o 
amdgpu_pll.o \

-    amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
+    amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o 
amdgpu_ctx_workload.o amdgpu_sync.o \
  amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o 
amdgpu_virt.o \

  amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
  amdgpu_debugfs.o amdgpu_ids.o amdgpu_gmc.o \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c

new file mode 100644
index ..a11cf29bc388
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
@@ -0,0 +1,97 @@
+/*
+ * Copyright 2022 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person 
obtaining a
+ * copy of this software and associated documentation files (the 
"Software"),
+ * to deal in the Software without restriction, including without 
limitation
+ * the rights to use, copy, modify, merge, publish, distribute, 
sublicense,

+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be 
included in

+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 
EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 
MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT 
SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, 
DAMAGES OR

+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#include 
+#include "kgd_pp_interface.h"
+#include "amdgpu_ctx_workload.h"
+
+static enum PP_SMC_POWER_PROFILE
+amdgpu_workload_to_power_profile(uint32_t hint)
+{
+    switch (hint) {
+    case AMDGPU_CTX_WORKLOAD_HINT_NONE:
+    default:
+    return PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT;
+
+    case AMDGPU_CTX_WORKLOAD_HINT_3D:
+    return PP_SMC_POWER_PROFILE_FULLSCREEN3D;
+    case AMDGPU_CTX_WORKLOAD_HINT_VIDEO:
+    return PP_SMC_POWER_PROFILE_VIDEO;
+    case AMDGPU_CTX_WORKLOAD_HINT_VR:
+    return PP_SMC_POWER_PROFILE_VR;
+    case AMDGPU_CTX_WORKLOAD_HINT_COMPUTE:
+    return PP_SMC_POWER_PROFILE_COMPUTE;
+    }
+}
+
+int amdgpu_set_workload_profile(struct amdgpu_device *adev,
+    uint32_t hint)
+{
+    int ret = 0;
+    enum PP_SMC_POWER_PROFILE profile =
+    amdgpu_workload_to_power_profile(hint);
+
+    if (adev->pm.workload_mode == hint)
+    return 0;
+
+    mutex_lock(>pm.smu_workload_lock);


If it's all about pm subsystem variable accesses, this API should rather 
be inside amd/pm subsystem. No need to expose the variable outside pm 
subsytem. Also currently all amdgpu_dpm* calls are protected under one 
mutex. Then this extra lock won't be needed.




This is tricky, this is not all about PM subsystem. Note that the job 
management and scheduling is handled into amdgpu_ctx, so the workload 
hint is set in context_management API. The API is consumed when the job 
is actually run from amdgpu_run() layer. So its a joint interface 
between context and PM.



+
+    if (adev->pm.workload_mode == hint)
+    goto unlock;
+
+    ret = amdgpu_dpm_switch_power_profile(adev, profile, 1);
+    if (!ret)
+    adev->pm.workload_mode = hint;
+    atomic_inc(>pm.workload_switch_ref);


Why is this reference kept? The swtiching happens inside a lock and 
there is already a check not to switch if the hint matches with current 
workload.




This reference is kept so that we would not reset the PM mode to 

Re: [PATCH v3 2/5] drm/amdgpu: add new functions to set GPU power profile

[Sharma, Shashank]

On 9/27/2022 11:29 AM, Quan, Evan wrote:

[AMD Official Use Only - General]




-Original Message-
From: Sharma, Shashank 
Sent: Tuesday, September 27, 2022 3:30 PM
To: Quan, Evan ; amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander ; Somalapuram,
Amaranath ; Koenig, Christian

Subject: Re: [PATCH v3 2/5] drm/amdgpu: add new functions to set GPU
power profile

Hello Evan,

On 9/27/2022 4:14 AM, Quan, Evan wrote:

[AMD Official Use Only - General]




-Original Message-
From: amd-gfx  On Behalf Of
Shashank Sharma
Sent: Tuesday, September 27, 2022 5:40 AM
To: amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander ; Somalapuram,
Amaranath ; Koenig, Christian
; Sharma, Shashank

Subject: [PATCH v3 2/5] drm/amdgpu: add new functions to set GPU
power profile

This patch adds new functions which will allow a user to change the
GPU power profile based a GPU workload hint flag.

Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
   drivers/gpu/drm/amd/amdgpu/Makefile   |  2 +-
   .../gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c  | 97
+++
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c|  1 +
   .../gpu/drm/amd/include/amdgpu_ctx_workload.h | 54 +++
   drivers/gpu/drm/amd/pm/inc/amdgpu_dpm.h   |  5 +
   5 files changed, 158 insertions(+), 1 deletion(-)
   create mode 100644
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
   create mode 100644
drivers/gpu/drm/amd/include/amdgpu_ctx_workload.h

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile
b/drivers/gpu/drm/amd/amdgpu/Makefile
index 5a283d12f8e1..34679c657ecc 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -50,7 +50,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o
amdgpu_pll.o \
-   amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
+   amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o
amdgpu_ctx_workload.o amdgpu_sync.o \
amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o
amdgpu_virt.o \
amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
amdgpu_debugfs.o amdgpu_ids.o amdgpu_gmc.o \ diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
new file mode 100644
index ..a11cf29bc388
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
@@ -0,0 +1,97 @@
+/*
+ * Copyright 2022 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person
+obtaining a
+ * copy of this software and associated documentation files (the
"Software"),
+ * to deal in the Software without restriction, including without
+ limitation
+ * the rights to use, copy, modify, merge, publish, distribute,
+ sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom
+ the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY

KIND,

EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN

NO

EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM,
DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR
THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#include 
+#include "kgd_pp_interface.h"
+#include "amdgpu_ctx_workload.h"
+
+static enum PP_SMC_POWER_PROFILE
+amdgpu_workload_to_power_profile(uint32_t hint) {
+   switch (hint) {
+   case AMDGPU_CTX_WORKLOAD_HINT_NONE:
+   default:
+   return PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT;
+
+   case AMDGPU_CTX_WORKLOAD_HINT_3D:
+   return PP_SMC_POWER_PROFILE_FULLSCREEN3D;
+   case AMDGPU_CTX_WORKLOAD_HINT_VIDEO:
+   return PP_SMC_POWER_PROFILE_VIDEO;
+   case AMDGPU_CTX_WORKLOAD_HINT_VR:
+   return PP_SMC_POWER_PROFILE_VR;
+   case AMDGPU_CTX_WORKLOAD_HINT_COMPUTE:
+   return PP_SMC_POWER_PROFILE_COMPUTE;
+   }
+}
+
+int amdgpu_set_workload_profile(struct amdgpu_device *adev,
+   uint32_t hint)
+{
+   int ret = 0;
+   enum PP_SMC_POWER_PROFILE profile =
+   amdgpu_workload_to_power_profile(hint);
+
+   if (adev->pm.workload_mode == hint)
+   return 0;
+
+   mutex_lock(>pm.smu_workload_lock);
+
+   if (adev->pm.workload_mode == hint)
+   goto unlock;

[Quan, Evan] This seems redundant with code above. I saw you dropped

this in Patch4.

But 

Re: [PATCH v3 2/5] drm/amdgpu: add new functions to set GPU power profile

[Sharma, Shashank]

Hello Evan,

On 9/27/2022 4:14 AM, Quan, Evan wrote:

[AMD Official Use Only - General]




-Original Message-
From: amd-gfx  On Behalf Of
Shashank Sharma
Sent: Tuesday, September 27, 2022 5:40 AM
To: amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander ; Somalapuram,
Amaranath ; Koenig, Christian
; Sharma, Shashank

Subject: [PATCH v3 2/5] drm/amdgpu: add new functions to set GPU power
profile

This patch adds new functions which will allow a user to
change the GPU power profile based a GPU workload hint
flag.

Cc: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/amd/amdgpu/Makefile   |  2 +-
  .../gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c  | 97
+++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c|  1 +
  .../gpu/drm/amd/include/amdgpu_ctx_workload.h | 54 +++
  drivers/gpu/drm/amd/pm/inc/amdgpu_dpm.h   |  5 +
  5 files changed, 158 insertions(+), 1 deletion(-)
  create mode 100644
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
  create mode 100644
drivers/gpu/drm/amd/include/amdgpu_ctx_workload.h

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile
b/drivers/gpu/drm/amd/amdgpu/Makefile
index 5a283d12f8e1..34679c657ecc 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -50,7 +50,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \
atombios_dp.o amdgpu_afmt.o amdgpu_trace_points.o \
atombios_encoders.o amdgpu_sa.o atombios_i2c.o \
amdgpu_dma_buf.o amdgpu_vm.o amdgpu_vm_pt.o amdgpu_ib.o
amdgpu_pll.o \
-   amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
+   amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o
amdgpu_ctx_workload.o amdgpu_sync.o \
amdgpu_gtt_mgr.o amdgpu_preempt_mgr.o amdgpu_vram_mgr.o
amdgpu_virt.o \
amdgpu_atomfirmware.o amdgpu_vf_error.o amdgpu_sched.o \
amdgpu_debugfs.o amdgpu_ids.o amdgpu_gmc.o \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
new file mode 100644
index ..a11cf29bc388
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx_workload.c
@@ -0,0 +1,97 @@
+/*
+ * Copyright 2022 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the
"Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO
EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM,
DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR
THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#include 
+#include "kgd_pp_interface.h"
+#include "amdgpu_ctx_workload.h"
+
+static enum PP_SMC_POWER_PROFILE
+amdgpu_workload_to_power_profile(uint32_t hint)
+{
+   switch (hint) {
+   case AMDGPU_CTX_WORKLOAD_HINT_NONE:
+   default:
+   return PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT;
+
+   case AMDGPU_CTX_WORKLOAD_HINT_3D:
+   return PP_SMC_POWER_PROFILE_FULLSCREEN3D;
+   case AMDGPU_CTX_WORKLOAD_HINT_VIDEO:
+   return PP_SMC_POWER_PROFILE_VIDEO;
+   case AMDGPU_CTX_WORKLOAD_HINT_VR:
+   return PP_SMC_POWER_PROFILE_VR;
+   case AMDGPU_CTX_WORKLOAD_HINT_COMPUTE:
+   return PP_SMC_POWER_PROFILE_COMPUTE;
+   }
+}
+
+int amdgpu_set_workload_profile(struct amdgpu_device *adev,
+   uint32_t hint)
+{
+   int ret = 0;
+   enum PP_SMC_POWER_PROFILE profile =
+   amdgpu_workload_to_power_profile(hint);
+
+   if (adev->pm.workload_mode == hint)
+   return 0;
+
+   mutex_lock(>pm.smu_workload_lock);
+
+   if (adev->pm.workload_mode == hint)
+   goto unlock;

[Quan, Evan] This seems redundant with code above. I saw you dropped this in 
Patch4.
But I kind of feel this should be the one which needs to be kept.


Yes, this shuffle happened during the rebase-testing of V3, will update 
this.



+
+   ret = amdgpu_dpm_switch_power_profile(adev, profile, 1);
+   if (!ret)
+   adev->pm.workload_mode = hint;
+   atomic_inc(>pm.workload_switch_ref);
+
+unlock:
+   mutex_unlock(>pm.smu_workload_lock);
+   

Re: [PATCH v2 1/5] drm/amdgpu: add UAPI for workload hints to ctx ioctl

[Sharma, Shashank]

On 9/26/2022 5:19 PM, Christian König wrote:

Am 26.09.22 um 17:14 schrieb Sharma, Shashank:


Hello Christian,

On 9/26/2022 5:10 PM, Christian König wrote:

Am 26.09.22 um 17:02 schrieb Shashank Sharma:

Allow the user to specify a workload hint to the kernel.
We can use these to tweak the dpm heuristics to better match
the workload for improved performance.

Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  include/uapi/drm/amdgpu_drm.h | 18 ++
  1 file changed, 18 insertions(+)

diff --git a/include/uapi/drm/amdgpu_drm.h 
b/include/uapi/drm/amdgpu_drm.h

index c2c9c674a223..da5019a6e233 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -212,6 +212,8 @@ union drm_amdgpu_bo_list {
  #define AMDGPU_CTX_OP_QUERY_STATE2    4
  #define AMDGPU_CTX_OP_GET_STABLE_PSTATE    5
  #define AMDGPU_CTX_OP_SET_STABLE_PSTATE    6
+#define AMDGPU_CTX_OP_GET_WORKLOAD_PROFILE    7


Do we really have an use case for getting the profile or is that just 
added for completeness?
To be honest, there is no direct use case for a get(). We have 
self-reset in kernel to clear the profile, so this is just for the 
sake of completeness.


The problem is we usually need an userspace use case to justify 
upstreaming of an UAPI.


We already had a couple of cases where we only upstreamed UAPI for the 
sake of completeness (set/get pair for example) and then years later 
found out that the getter is completely broken for years because nobody 
used it.


So if we don't really need it I would try to avoid it.

Christian.


Makes sense, I can remove get API as UAPI and just keep it kernel internal.

- Shashank






At base minimum we need a test-case for both to exercise the UAPI.



Agree, I have already added some test cases in libdrm/tests/amdgpu for 
my local testing, I will start publishing them once we have an 
agreement on the UAPI and kernel design.


- Shashank


Regards,
Christian.


+#define AMDGPU_CTX_OP_SET_WORKLOAD_PROFILE    8
  /* GPU reset status */
  #define AMDGPU_CTX_NO_RESET    0
@@ -252,6 +254,17 @@ union drm_amdgpu_bo_list {
  #define AMDGPU_CTX_STABLE_PSTATE_MIN_MCLK  3
  #define AMDGPU_CTX_STABLE_PSTATE_PEAK  4
+/* GPU workload hints, flag bits 8-15 */
+#define AMDGPU_CTX_WORKLOAD_HINT_SHIFT 8
+#define AMDGPU_CTX_WORKLOAD_HINT_MASK  (0xff << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_NONE  (0 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_3D    (1 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_VIDEO (2 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_VR    (3 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_COMPUTE   (4 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)

+#define AMDGPU_CTX_WORKLOAD_HINT_MAX AMDGPU_CTX_WORKLOAD_HINT_COMPUTE
+#define AMDGPU_CTX_WORKLOAD_INDEX(n)   (n >> 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)

+
  struct drm_amdgpu_ctx_in {
  /** AMDGPU_CTX_OP_* */
  __u32    op;
@@ -281,6 +294,11 @@ union drm_amdgpu_ctx_out {
  __u32    flags;
  __u32    _pad;
  } pstate;
+
+    struct {
+    __u32    flags;
+    __u32    _pad;
+    } workload;
  };
  union drm_amdgpu_ctx {

Re: [PATCH v2 1/5] drm/amdgpu: add UAPI for workload hints to ctx ioctl

[Sharma, Shashank]

Hello Christian,

On 9/26/2022 5:10 PM, Christian König wrote:

Am 26.09.22 um 17:02 schrieb Shashank Sharma:

Allow the user to specify a workload hint to the kernel.
We can use these to tweak the dpm heuristics to better match
the workload for improved performance.

Signed-off-by: Alex Deucher 
Signed-off-by: Shashank Sharma 
---
  include/uapi/drm/amdgpu_drm.h | 18 ++
  1 file changed, 18 insertions(+)

diff --git a/include/uapi/drm/amdgpu_drm.h 
b/include/uapi/drm/amdgpu_drm.h

index c2c9c674a223..da5019a6e233 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -212,6 +212,8 @@ union drm_amdgpu_bo_list {
  #define AMDGPU_CTX_OP_QUERY_STATE2    4
  #define AMDGPU_CTX_OP_GET_STABLE_PSTATE    5
  #define AMDGPU_CTX_OP_SET_STABLE_PSTATE    6
+#define AMDGPU_CTX_OP_GET_WORKLOAD_PROFILE    7


Do we really have an use case for getting the profile or is that just 
added for completeness?
To be honest, there is no direct use case for a get(). We have 
self-reset in kernel to clear the profile, so this is just for the sake 
of completeness.



At base minimum we need a test-case for both to exercise the UAPI.



Agree, I have already added some test cases in libdrm/tests/amdgpu for 
my local testing, I will start publishing them once we have an agreement 
on the UAPI and kernel design.


- Shashank


Regards,
Christian.


+#define AMDGPU_CTX_OP_SET_WORKLOAD_PROFILE    8
  /* GPU reset status */
  #define AMDGPU_CTX_NO_RESET    0
@@ -252,6 +254,17 @@ union drm_amdgpu_bo_list {
  #define AMDGPU_CTX_STABLE_PSTATE_MIN_MCLK  3
  #define AMDGPU_CTX_STABLE_PSTATE_PEAK  4
+/* GPU workload hints, flag bits 8-15 */
+#define AMDGPU_CTX_WORKLOAD_HINT_SHIFT 8
+#define AMDGPU_CTX_WORKLOAD_HINT_MASK  (0xff << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_NONE  (0 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_3D    (1 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_VIDEO (2 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_VR    (3 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_COMPUTE   (4 << 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)
+#define AMDGPU_CTX_WORKLOAD_HINT_MAX   
AMDGPU_CTX_WORKLOAD_HINT_COMPUTE
+#define AMDGPU_CTX_WORKLOAD_INDEX(n)   (n >> 
AMDGPU_CTX_WORKLOAD_HINT_SHIFT)

+
  struct drm_amdgpu_ctx_in {
  /** AMDGPU_CTX_OP_* */
  __u32    op;
@@ -281,6 +294,11 @@ union drm_amdgpu_ctx_out {
  __u32    flags;
  __u32    _pad;
  } pstate;
+
+    struct {
+    __u32    flags;
+    __u32    _pad;
+    } workload;
  };
  union drm_amdgpu_ctx {

Re: [PATCH] amdgpu: add context creation flags in CS IOCTL

[Sharma, Shashank]

On 8/8/2022 12:59 PM, Christian König wrote:

Am 02.08.22 um 15:55 schrieb Shashank Sharma:

This patch adds:
- A new input parameter "flags" in the amdgpu_ctx_create2 call.
- Some new flags defining workload type hints.
- Some change in the caller function of amdgpu_ctx_create2, to
   accomodate this new parameter.

The idea is to pass the workload hints while context creation,


Bad idea.

Please take AMDGPU_CTX_OP_SET_STABLE_PSTATE and 
AMDGPU_CTX_OP_GET_STABLE_PSTATE as blueprint for this and don't add 
extra flags to the context creation.


Regards,
Christian.


Hey Christian,
Noted, let me have a look at AMDGPU_CTX_OP_GET/SET_STABLE_PSTATE 
implementation.


- Shashank





  so
that kernel GPU scheduler can pass this information to GPU FW, which in
turn can adjust the GPU characterstics as per the workload type.

Signed-off-by: Shashank Sharma 
Cc: Alex Deucher 
Cc: Marek Olsak 
Cc: Christian Koenig 
Cc: Amarnath Somalapuram 
---
  amdgpu/amdgpu.h  |  2 ++
  amdgpu/amdgpu_cs.c   |  5 -
  include/drm/amdgpu_drm.h | 10 +-
  3 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/amdgpu/amdgpu.h b/amdgpu/amdgpu.h
index b118dd48..1ebb46e6 100644
--- a/amdgpu/amdgpu.h
+++ b/amdgpu/amdgpu.h
@@ -874,6 +874,7 @@ int amdgpu_bo_list_update(amdgpu_bo_list_handle 
handle,

   *
   * \param   dev  - \c [in] Device handle. See 
#amdgpu_device_initialize()
   * \param   priority - \c [in] Context creation flags. See 
AMDGPU_CTX_PRIORITY_*

+ * \param   flags    - \c [in] Context flags. See AMDGPU_CTX_FLAGS_*
   * \param   context  - \c [out] GPU Context handle
   *
   * \return   0 on success\n
@@ -884,6 +885,7 @@ int amdgpu_bo_list_update(amdgpu_bo_list_handle 
handle,

  */
  int amdgpu_cs_ctx_create2(amdgpu_device_handle dev,
   uint32_t priority,
+ uint32_t flags,
   amdgpu_context_handle *context);
  /**
   * Create GPU execution Context
diff --git a/amdgpu/amdgpu_cs.c b/amdgpu/amdgpu_cs.c
index fad484bf..d4723ea5 100644
--- a/amdgpu/amdgpu_cs.c
+++ b/amdgpu/amdgpu_cs.c
@@ -44,12 +44,14 @@ static int 
amdgpu_cs_reset_sem(amdgpu_semaphore_handle sem);

   *
   * \param   dev  - \c [in] Device handle. See 
#amdgpu_device_initialize()
   * \param   priority - \c [in] Context creation flags. See 
AMDGPU_CTX_PRIORITY_*

+ * \param   flags    - \c [in] Context flags. See AMDGPU_CTX_FLAGS_*
   * \param   context  - \c [out] GPU Context handle
   *
   * \return  0 on success otherwise POSIX Error code
  */
  drm_public int amdgpu_cs_ctx_create2(amdgpu_device_handle dev,
   uint32_t priority,
+ uint32_t flags,
   amdgpu_context_handle *context)
  {
  struct amdgpu_context *gpu_context;
@@ -74,6 +76,7 @@ drm_public int 
amdgpu_cs_ctx_create2(amdgpu_device_handle dev,

  memset(, 0, sizeof(args));
  args.in.op = AMDGPU_CTX_OP_ALLOC_CTX;
  args.in.priority = priority;
+    args.in.flags = flags;
  r = drmCommandWriteRead(dev->fd, DRM_AMDGPU_CTX, , 
sizeof(args));

  if (r)
@@ -97,7 +100,7 @@ error:
  drm_public int amdgpu_cs_ctx_create(amdgpu_device_handle dev,
  amdgpu_context_handle *context)
  {
-    return amdgpu_cs_ctx_create2(dev, AMDGPU_CTX_PRIORITY_NORMAL, 
context);
+    return amdgpu_cs_ctx_create2(dev, AMDGPU_CTX_PRIORITY_NORMAL, 0, 
context);

  }
  /**
diff --git a/include/drm/amdgpu_drm.h b/include/drm/amdgpu_drm.h
index 0cbd1540..d9fb1f20 100644
--- a/include/drm/amdgpu_drm.h
+++ b/include/drm/amdgpu_drm.h
@@ -238,10 +238,18 @@ union drm_amdgpu_bo_list {
  #define AMDGPU_CTX_PRIORITY_HIGH    512
  #define AMDGPU_CTX_PRIORITY_VERY_HIGH   1023
+/* GPU context workload hint bitmask */
+#define AMDGPU_CTX_FLAGS_WORKLOAD_HINT_MASK    0xFF
+#define AMDGPU_CTX_FLAGS_WORKLOAD_HINT_NONE    0
+#define AMDGPU_CTX_FLAGS_WORKLOAD_HINT_3D  (1 << 1)
+#define AMDGPU_CTX_FLAGS_WORKLOAD_HINT_VIDEO   (1 << 2)
+#define AMDGPU_CTX_FLAGS_WORKLOAD_HINT_VR  (1 << 3)
+#define AMDGPU_CTX_FLAGS_WORKLOAD_HINT_COMPUTE (1 << 4)
+
  struct drm_amdgpu_ctx_in {
  /** AMDGPU_CTX_OP_* */
  __u32    op;
-    /** For future use, no flags defined so far */
+    /** AMDGPU_CTX_FLAGS_* */
  __u32    flags;
  __u32    ctx_id;
  /** AMDGPU_CTX_PRIORITY_* */

Re: [PATCH] amdgpu: add context creation flags in CS IOCTL

[Sharma, Shashank]

On 8/2/2022 5:58 PM, Michel Dänzer wrote:

On 2022-08-02 15:55, Shashank Sharma wrote:

This patch adds:
- A new input parameter "flags" in the amdgpu_ctx_create2 call.
- Some new flags defining workload type hints.
- Some change in the caller function of amdgpu_ctx_create2, to
   accomodate this new parameter.

The idea is to pass the workload hints while context creation, so
that kernel GPU scheduler can pass this information to GPU FW, which in
turn can adjust the GPU characterstics as per the workload type.

Signed-off-by: Shashank Sharma 
Cc: Alex Deucher 
Cc: Marek Olsak 
Cc: Christian Koenig 
Cc: Amarnath Somalapuram 
---
  amdgpu/amdgpu.h  |  2 ++
  amdgpu/amdgpu_cs.c   |  5 -
  include/drm/amdgpu_drm.h | 10 +-


See include/drm/README on how to handle changes to include/drm/*_drm.h .




Also, libdrm changes are now reviewed and merged as GitLab merge requests: 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fmesa%2Fdrm%2F-%2Fmerge_requestsdata=05%7C01%7Cshashank.sharma%40amd.com%7C50e342a42eac4d4d4fd408da749fd574%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637950527087917031%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=qU41z3Hxo0jKMWgehDWLYi7r4x1QDsA%2F2KfOigj9iew%3Dreserved=0



Noted, Thanks.

Regards
Shashank

RE: [PATCH v3 2/2] drm/amdgpu: adding device coredump support

[Sharma, Shashank]

Reviewed-by: Shashank Sharma  for both the patches. 

Regards
Shashank
-Original Message-
From: Somalapuram, Amaranath  
Sent: Thursday, June 2, 2022 10:16 AM
To: amd-gfx@lists.freedesktop.org
Cc: Koenig, Christian ; Deucher, Alexander 
; Sharma, Shashank ; 
Somalapuram, Amaranath 
Subject: [PATCH v3 2/2] drm/amdgpu: adding device coredump support

Added device coredump information:
- Kernel version
- Module
- Time
- VRAM status
- Guilty process name and PID
- GPU register dumps
v1 -> v2: Variable name change
v1 -> v2: NULL check
v1 -> v2: Code alignment
v1 -> v2: Adding dummy amdgpu_devcoredump_free
v1 -> v2: memset reset_task_info to zero
v2 -> v3: add CONFIG_DEV_COREDUMP for variables
v2 -> v3: remove NULL check on amdgpu_devcoredump_read

Signed-off-by: Somalapuram Amaranath 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h|  5 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 64 ++
 2 files changed, 69 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index c79d9992b113..1bfbaf65d414 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1044,6 +1044,11 @@ struct amdgpu_device {
uint32_t*reset_dump_reg_list;
uint32_t*reset_dump_reg_value;
int num_regs;
+#ifdef CONFIG_DEV_COREDUMP
+   struct amdgpu_task_info reset_task_info;
+   boolreset_vram_lost;
+   struct timespec64   reset_time;
+#endif
 
struct amdgpu_reset_domain  *reset_domain;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 89c6db03e84b..f1def74aaad0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -32,6 +32,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 
 #include 
 #include 
@@ -4734,6 +4736,59 @@ static int amdgpu_reset_reg_dumps(struct amdgpu_device 
*adev)
return 0;
 }
 
+#ifdef CONFIG_DEV_COREDUMP
+static ssize_t amdgpu_devcoredump_read(char *buffer, loff_t offset,
+   size_t count, void *data, size_t datalen) {
+   struct drm_printer p;
+   struct amdgpu_device *adev = data;
+   struct drm_print_iterator iter;
+   int i;
+
+   iter.data = buffer;
+   iter.offset = 0;
+   iter.start = offset;
+   iter.remain = count;
+
+   p = drm_coredump_printer();
+
+   drm_printf(, " AMDGPU Device Coredump \n");
+   drm_printf(, "kernel: " UTS_RELEASE "\n");
+   drm_printf(, "module: " KBUILD_MODNAME "\n");
+   drm_printf(, "time: %lld.%09ld\n", adev->reset_time.tv_sec, 
adev->reset_time.tv_nsec);
+   if (adev->reset_task_info.pid)
+   drm_printf(, "process_name: %s PID: %d\n",
+  adev->reset_task_info.process_name,
+  adev->reset_task_info.pid);
+
+   if (adev->reset_vram_lost)
+   drm_printf(, "VRAM is lost due to GPU reset!\n");
+   if (adev->num_regs) {
+   drm_printf(, "AMDGPU register dumps:\nOffset: Value:\n");
+
+   for (i = 0; i < adev->num_regs; i++)
+   drm_printf(, "0x%08x: 0x%08x\n",
+  adev->reset_dump_reg_list[i],
+  adev->reset_dump_reg_value[i]);
+   }
+
+   return count - iter.remain;
+}
+
+static void amdgpu_devcoredump_free(void *data) { }
+
+static void amdgpu_reset_capture_coredumpm(struct amdgpu_device *adev) 
+{
+   struct drm_device *dev = adev_to_drm(adev);
+
+   ktime_get_ts64(>reset_time);
+   dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_KERNEL,
+ amdgpu_devcoredump_read, amdgpu_devcoredump_free); } 
#endif
+
 int amdgpu_do_asic_reset(struct list_head *device_list_handle,
 struct amdgpu_reset_context *reset_context)  { @@ 
-4818,6 +4873,15 @@ int amdgpu_do_asic_reset(struct list_head 
*device_list_handle,
goto out;
 
vram_lost = 
amdgpu_device_check_vram_lost(tmp_adev);
+#ifdef CONFIG_DEV_COREDUMP
+   tmp_adev->reset_vram_lost = vram_lost;
+   memset(_adev->reset_task_info, 0,
+   
sizeof(tmp_adev->reset_task_info));
+   if (reset_context->job && 
reset_context->job->vm)
+   tmp_adev->reset_task_info =
+   
reset_context->job->vm->task_info;
+   

RE: [PATCH v2 1/2] drm/amdgpu: save the reset dump register value for devcoredump

[Sharma, Shashank]

It’s not correct even in the snapshot .

[cid:image001.png@01D87501.96BF5BD0]

The *a*dev->reset_dump line should start just below the open brace in the line 
above “dumps*(*adev->reset”.

Check the Linux kernel coding guidelines for extending the line below the open 
brace.


Regards
Shashank
From: Somalapuram, Amaranath 
Sent: Tuesday, May 31, 2022 3:09 PM
To: Sharma, Shashank ; Somalapuram, Amaranath 
; amd-gfx@lists.freedesktop.org
Cc: Koenig, Christian ; Deucher, Alexander 

Subject: Re: [PATCH v2 1/2] drm/amdgpu: save the reset dump register value for 
devcoredump



On 5/26/2022 3:24 PM, Sharma, Shashank wrote:
Hey Amar,

On 5/26/2022 11:48 AM, Somalapuram Amaranath wrote:

Allocate memory for register value and use the same values for devcoredump.
v1 -> V2: Change krealloc_array() to kmalloc_array()

Signed-off-by: Somalapuram Amaranath 
<mailto:amaranath.somalapu...@amd.com>
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 7 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  | 6 +++---
  3 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 76df583663c7..c79d9992b113 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1042,6 +1042,7 @@ struct amdgpu_device {
/* reset dump register */
  uint32_t*reset_dump_reg_list;
+uint32_t*reset_dump_reg_value;
  int num_regs;
struct amdgpu_reset_domain*reset_domain;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index eedb12f6b8a3..f3ac7912c29c 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
@@ -1709,17 +1709,24 @@ static ssize_t 
amdgpu_reset_dump_register_list_write(struct file *f,
  i++;
  } while (len < size);
  +new = kmalloc_array(i, sizeof(uint32_t), GFP_KERNEL);
+if (!new) {
+ret = -ENOMEM;
+goto error_free;
+}
  ret = down_write_killable(>reset_domain->sem);
  if (ret)
  goto error_free;
swap(adev->reset_dump_reg_list, tmp);
+swap(adev->reset_dump_reg_value, new);
  adev->num_regs = i;
  up_write(>reset_domain->sem);
  ret = size;
error_free:
  kfree(tmp);
+kfree(new);
  return ret;
  }
  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 4daa0e893965..866b4980a6fa 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -4720,15 +4720,15 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device 
*adev,
static int amdgpu_reset_reg_dumps(struct amdgpu_device *adev)
  {
-uint32_t reg_value;
  int i;
lockdep_assert_held(>reset_domain->sem);
  dump_stack();
for (i = 0; i < adev->num_regs; i++) {
-reg_value = RREG32(adev->reset_dump_reg_list[i]);
-trace_amdgpu_reset_reg_dumps(adev->reset_dump_reg_list[i], reg_value);
+adev->reset_dump_reg_value[i] = RREG32(adev->reset_dump_reg_list[i]);
+trace_amdgpu_reset_reg_dumps(adev->reset_dump_reg_list[i],
+adev->reset_dump_reg_value[i]);

Alignment is showing spaces in email. But its 2 tabs in code.

Regards,
S.Amarnath


Please fix the alignment with the line above, after that, please feel free to 
use:
Reviewed-by: Shashank Sharma 
<mailto:shashank.sha...@amd.com>

- Shashank

  }
return 0;

Re: [PATCH v2 2/2] drm/amdgpu: adding device coredump support

[Sharma, Shashank]

On 5/26/2022 11:48 AM, Somalapuram Amaranath wrote:

Added device coredump information:
- Kernel version
- Module
- Time
- VRAM status
- Guilty process name and PID
- GPU register dumps
v1 -> v2: Variable name change
v1 -> v2: NULL check
v1 -> v2: Code alignment
v1 -> v2: Adding dummy amdgpu_devcoredump_free
v1 -> v2: memset reset_task_info to zero

Signed-off-by: Somalapuram Amaranath 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h|  3 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 67 ++
  2 files changed, 70 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index c79d9992b113..25a7b2c74928 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1044,6 +1044,9 @@ struct amdgpu_device {
uint32_t*reset_dump_reg_list;
uint32_t*reset_dump_reg_value;
int num_regs;
+   struct amdgpu_task_info reset_task_info;
+   boolreset_vram_lost;
+   struct timespec64   reset_time;
  
  	struct amdgpu_reset_domain	*reset_domain;
  
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

index 866b4980a6fa..ca97afe5be63 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -32,6 +32,8 @@
  #include 
  #include 
  #include 
+#include 
+#include 
  
  #include 

  #include 
@@ -4734,6 +4736,62 @@ static int amdgpu_reset_reg_dumps(struct amdgpu_device 
*adev)
return 0;
  }
  
+#ifdef CONFIG_DEV_COREDUMP

+static ssize_t amdgpu_devcoredump_read(char *buffer, loff_t offset,
+   size_t count, void *data, size_t datalen)
+{
+   struct drm_printer p;
+   struct amdgpu_device *adev = data;
+   struct drm_print_iterator iter;
+   int i;
+
+   if (adev == NULL)
+   return 0;
+
+   iter.data = buffer;
+   iter.offset = 0;
+   iter.start = offset;
+   iter.remain = count;
+
+   p = drm_coredump_printer();
+
+   drm_printf(, " AMDGPU Device Coredump \n");
+   drm_printf(, "kernel: " UTS_RELEASE "\n");
+   drm_printf(, "module: " KBUILD_MODNAME "\n");
+   drm_printf(, "time: %lld.%09ld\n", adev->reset_time.tv_sec, 
adev->reset_time.tv_nsec);
+   if (adev->reset_task_info.pid)
+   drm_printf(, "process_name: %s PID: %d\n",
+  adev->reset_task_info.process_name,
+  adev->reset_task_info.pid);
+
+   if (adev->reset_vram_lost)
+   drm_printf(, "VRAM is lost due to GPU reset!\n");
+   if (adev->num_regs) {
+   drm_printf(, "AMDGPU register dumps:\nOffset: Value:\n");
+
+   for (i = 0; i < adev->num_regs; i++)
+   drm_printf(, "0x%08x: 0x%08x\n",
+  adev->reset_dump_reg_list[i],
+  adev->reset_dump_reg_value[i]);
+   }
+
+   return count - iter.remain;
+}
+
+static void amdgpu_devcoredump_free(void *data)
+{
+}
+
+static void amdgpu_reset_capture_coredumpm(struct amdgpu_device *adev)
+{
+   struct drm_device *dev = adev_to_drm(adev);
+
+   ktime_get_ts64(>reset_time);
+   dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_KERNEL,
+   amdgpu_devcoredump_read, amdgpu_devcoredump_free);

Alignment with line above.

+}
+#endif
+
  int amdgpu_do_asic_reset(struct list_head *device_list_handle,
 struct amdgpu_reset_context *reset_context)
  {
@@ -4818,6 +4876,15 @@ int amdgpu_do_asic_reset(struct list_head 
*device_list_handle,
goto out;
  
  vram_lost = amdgpu_device_check_vram_lost(tmp_adev);

+#ifdef CONFIG_DEV_COREDUMP
+   tmp_adev->reset_vram_lost = vram_lost;
+   memset(_adev->reset_task_info, 0,
+   
sizeof(tmp_adev->reset_task_info));

Alignment with the line above.

+   if (reset_context->job && 
reset_context->job->vm)
+   tmp_adev->reset_task_info =
+   
reset_context->job->vm->task_info;
+   amdgpu_reset_capture_coredumpm(tmp_adev);
+#endif
if (vram_lost) {
        DRM_INFO("VRAM is lost due to GPU 
reset!\n");
amdgpu_inc_vram_lost(tmp_adev);


With above fixed, feel free to use:
Reviewed-by: Shashank Sharma 

- Shashank

Re: [PATCH v2 1/2] drm/amdgpu: save the reset dump register value for devcoredump

[Sharma, Shashank]

Hey Amar,

On 5/26/2022 11:48 AM, Somalapuram Amaranath wrote:

Allocate memory for register value and use the same values for devcoredump.
v1 -> V2: Change krealloc_array() to kmalloc_array()

Signed-off-by: Somalapuram Amaranath 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 7 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  | 6 +++---
  3 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 76df583663c7..c79d9992b113 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1042,6 +1042,7 @@ struct amdgpu_device {
  
  	/* reset dump register */

uint32_t*reset_dump_reg_list;
+   uint32_t*reset_dump_reg_value;
int num_regs;
  
  	struct amdgpu_reset_domain	*reset_domain;

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index eedb12f6b8a3..f3ac7912c29c 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
@@ -1709,17 +1709,24 @@ static ssize_t 
amdgpu_reset_dump_register_list_write(struct file *f,
i++;
} while (len < size);
  
+	new = kmalloc_array(i, sizeof(uint32_t), GFP_KERNEL);

+   if (!new) {
+   ret = -ENOMEM;
+   goto error_free;
+   }
ret = down_write_killable(>reset_domain->sem);
if (ret)
goto error_free;
  
  	swap(adev->reset_dump_reg_list, tmp);

+   swap(adev->reset_dump_reg_value, new);
adev->num_regs = i;
up_write(>reset_domain->sem);
ret = size;
  
  error_free:

kfree(tmp);
+   kfree(new);
return ret;
  }
  
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

index 4daa0e893965..866b4980a6fa 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -4720,15 +4720,15 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device 
*adev,
  
  static int amdgpu_reset_reg_dumps(struct amdgpu_device *adev)

  {
-   uint32_t reg_value;
int i;
  
  	lockdep_assert_held(>reset_domain->sem);

dump_stack();
  
  	for (i = 0; i < adev->num_regs; i++) {

-   reg_value = RREG32(adev->reset_dump_reg_list[i]);
-   trace_amdgpu_reset_reg_dumps(adev->reset_dump_reg_list[i], 
reg_value);
+   adev->reset_dump_reg_value[i] = 
RREG32(adev->reset_dump_reg_list[i]);
+   trace_amdgpu_reset_reg_dumps(adev->reset_dump_reg_list[i],
+   adev->reset_dump_reg_value[i]);
Please fix the alignment with the line above, after that, please feel 
free to use:

Reviewed-by: Shashank Sharma 

- Shashank

}
  
  	return 0;

Re: [PATCH v1 2/2] drm/amdgpu: adding device coredump support

[Sharma, Shashank]

(snip)

#ifdef CONFIG_DEV_COREDUMP
     tmp_adev->reset_context_vram_lost = 
vram_lost;

tmp_adev->reset_context_task_info.pid = 0;
     if (reset_context->job && 
reset_context->job->vm)

tmp_adev->reset_context_task_info =
reset_context->job->vm->task_info;
amdgpu_reset_capture_coredumpm(tmp_adev);
#endif
tmp_adev->reset_context_task_info refers to 
reset_context->job->vm->task_info,
i am not setting PID and process name separately, instead i am having 
the reference to the reset_context->job->vm->task_info on valid VM.




I think you are not getting my point. struct amdgpu_task_info has 3 more 
parameters: process_name, task_name and tgid.


When VRAM is not lost: we are getting the whole new amdgpu_task_info 
structure, and all parameters are valid.


When VRAM is lost: we should reset all parameters of amdgpu_task_info 
structure, not just pid. Else other params may contain garbage.


So what I mean is:
instead of
tmp_adev->reset_context_task_info.pid = 0;

do something like:
memset(_adev->reset_context_task_info.pid, 0, 
sizeof(reset_context_task_info));


- Shashank

Re: [PATCH v1 2/2] drm/amdgpu: adding device coredump support

[Sharma, Shashank]

On 5/24/2022 3:18 PM, Somalapuram, Amaranath wrote:


On 5/24/2022 6:20 PM, Sharma, Shashank wrote:



On 5/24/2022 2:10 PM, Somalapuram, Amaranath wrote:


On 5/24/2022 3:23 PM, Sharma, Shashank wrote:



On 5/24/2022 8:42 AM, Somalapuram, Amaranath wrote:


On 5/20/2022 7:52 PM, Sharma, Shashank wrote:



On 5/20/2022 3:49 PM, Somalapuram Amaranath wrote:

Added device coredump information:
- Kernel version
- Module
- Time
- VRAM status
- Guilty process name and PID
- GPU register dumps

Signed-off-by: Somalapuram Amaranath 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h    |  3 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 59 
++

  2 files changed, 62 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu.h

index c79d9992b113..f28d9c563f74 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1044,6 +1044,9 @@ struct amdgpu_device {
  uint32_t *reset_dump_reg_list;
  uint32_t    *reset_dump_reg_value;
  int num_regs;
+    struct amdgpu_task_info reset_context_task_info;
+    bool reset_context_vram_lost;


How about drop the 'context' from name and just reset_task_info 
and reset_vram_lost ?

OK.



+    struct timespec64 reset_time;
    struct amdgpu_reset_domain    *reset_domain;
  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

index 963c897a76e6..f9b710e741a7 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -32,6 +32,8 @@
  #include 
  #include 
  #include 
+#include 
+#include 
    #include 
  #include 
@@ -4733,6 +4735,55 @@ static int amdgpu_reset_reg_dumps(struct 
amdgpu_device *adev)

  return 0;
  }
  +#ifdef CONFIG_DEV_COREDUMP
+static ssize_t amdgpu_devcoredump_read(char *buffer, loff_t offset,
+    size_t count, void *data, size_t datalen)
+{
+    struct drm_printer p;
+    struct amdgpu_device *adev = data;
+    struct drm_print_iterator iter;
+    int i;
+


A NULL check for 'buffer' here could prevent a segfault later.


Agreed.

+    iter.data = buffer;
+    iter.offset = 0;
+    iter.start = offset;
+    iter.remain = count;
+
+    p = drm_coredump_printer();
+
+    drm_printf(, " AMDGPU Device Coredump \n");
+    drm_printf(, "kernel: " UTS_RELEASE "\n");
+    drm_printf(, "module: " KBUILD_MODNAME "\n");
+    drm_printf(, "time: %lld.%09ld\n", 
adev->reset_time.tv_sec, adev->reset_time.tv_nsec);

+    if (adev->reset_context_task_info.pid)
+    drm_printf(, "process_name: %s PID: %d\n",
+ adev->reset_context_task_info.process_name,
+ adev->reset_context_task_info.pid);

Please fix the alignment of print variables.


I will cross check this.

+
+    if (adev->reset_context_vram_lost)
+    drm_printf(, "VRAM is lost due to GPU reset!\n");
+    if (adev->num_regs) {
+    drm_printf(, "AMDGPU register dumps:\nOffset: Value:\n");
+
+    for (i = 0; i < adev->num_regs; i++)
+    drm_printf(, "0x%08x: 0x%08x\n",
+    adev->reset_dump_reg_list[i],
+    adev->reset_dump_reg_value[i]);
+    }
+
+    return count - iter.remain;
+}
+
+static void amdgpu_reset_capture_coredumpm(struct amdgpu_device 
*adev)

+{
+    struct drm_device *dev = adev_to_drm(adev);
+
+    ktime_get_ts64(>reset_time);
+    dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_KERNEL,
+    amdgpu_devcoredump_read, NULL);
instead of registering NULL as free function, I would prefer you 
to have a dummy no_op free function registered, which we can 
consume if something changes.

you mean something like this (function without any code):
staticvoidamdgpu_devcoredump_free(void*data)
{
}


Yes, precisely.


+}
+#endif
+
  int amdgpu_do_asic_reset(struct list_head *device_list_handle,
   struct amdgpu_reset_context *reset_context)
  {
@@ -4817,6 +4868,14 @@ int amdgpu_do_asic_reset(struct list_head 
*device_list_handle,

  goto out;
    vram_lost = 
amdgpu_device_check_vram_lost(tmp_adev);

+#ifdef CONFIG_DEV_COREDUMP
+    tmp_adev->reset_context_vram_lost = vram_lost;
+ tmp_adev->reset_context_task_info.pid = 0;

why is the PID hardcoded to 0 ?
in case of reset context reset_context->job->vm  is null 
(possibility that reset can be non VM related).
If we don't set tmp_adev->reset_context_task_info.pid = 0, it will 
show previous reset valid PID.




But when the VM is not NULL, are we updating this PID somewhere ? I 
did not see that happening in this series.

This is the only place where PID get updated.
For example sequence of operation like:
1st reset:
-valid VM and tmp_adev->reset_context_task_info.pid is set to some 
valid PID

2nd reset:
-invalid VM
-tm

Re: [PATCH v1 2/2] drm/amdgpu: adding device coredump support

[Sharma, Shashank]

On 5/24/2022 2:10 PM, Somalapuram, Amaranath wrote:


On 5/24/2022 3:23 PM, Sharma, Shashank wrote:



On 5/24/2022 8:42 AM, Somalapuram, Amaranath wrote:


On 5/20/2022 7:52 PM, Sharma, Shashank wrote:



On 5/20/2022 3:49 PM, Somalapuram Amaranath wrote:

Added device coredump information:
- Kernel version
- Module
- Time
- VRAM status
- Guilty process name and PID
- GPU register dumps

Signed-off-by: Somalapuram Amaranath 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h    |  3 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 59 
++

  2 files changed, 62 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu.h

index c79d9992b113..f28d9c563f74 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1044,6 +1044,9 @@ struct amdgpu_device {
  uint32_t    *reset_dump_reg_list;
  uint32_t    *reset_dump_reg_value;
  int num_regs;
+    struct amdgpu_task_info reset_context_task_info;
+    bool reset_context_vram_lost;


How about drop the 'context' from name and just reset_task_info and 
reset_vram_lost ?

OK.



+    struct timespec64 reset_time;
    struct amdgpu_reset_domain    *reset_domain;
  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

index 963c897a76e6..f9b710e741a7 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -32,6 +32,8 @@
  #include 
  #include 
  #include 
+#include 
+#include 
    #include 
  #include 
@@ -4733,6 +4735,55 @@ static int amdgpu_reset_reg_dumps(struct 
amdgpu_device *adev)

  return 0;
  }
  +#ifdef CONFIG_DEV_COREDUMP
+static ssize_t amdgpu_devcoredump_read(char *buffer, loff_t offset,
+    size_t count, void *data, size_t datalen)
+{
+    struct drm_printer p;
+    struct amdgpu_device *adev = data;
+    struct drm_print_iterator iter;
+    int i;
+


A NULL check for 'buffer' here could prevent a segfault later.


Agreed.

+    iter.data = buffer;
+    iter.offset = 0;
+    iter.start = offset;
+    iter.remain = count;
+
+    p = drm_coredump_printer();
+
+    drm_printf(, " AMDGPU Device Coredump \n");
+    drm_printf(, "kernel: " UTS_RELEASE "\n");
+    drm_printf(, "module: " KBUILD_MODNAME "\n");
+    drm_printf(, "time: %lld.%09ld\n", adev->reset_time.tv_sec, 
adev->reset_time.tv_nsec);

+    if (adev->reset_context_task_info.pid)
+    drm_printf(, "process_name: %s PID: %d\n",
+ adev->reset_context_task_info.process_name,
+ adev->reset_context_task_info.pid);

Please fix the alignment of print variables.


I will cross check this.

+
+    if (adev->reset_context_vram_lost)
+    drm_printf(, "VRAM is lost due to GPU reset!\n");
+    if (adev->num_regs) {
+    drm_printf(, "AMDGPU register dumps:\nOffset: Value:\n");
+
+    for (i = 0; i < adev->num_regs; i++)
+    drm_printf(, "0x%08x: 0x%08x\n",
+    adev->reset_dump_reg_list[i],
+    adev->reset_dump_reg_value[i]);
+    }
+
+    return count - iter.remain;
+}
+
+static void amdgpu_reset_capture_coredumpm(struct amdgpu_device 
*adev)

+{
+    struct drm_device *dev = adev_to_drm(adev);
+
+    ktime_get_ts64(>reset_time);
+    dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_KERNEL,
+    amdgpu_devcoredump_read, NULL);
instead of registering NULL as free function, I would prefer you to 
have a dummy no_op free function registered, which we can consume if 
something changes.

you mean something like this (function without any code):
staticvoidamdgpu_devcoredump_free(void*data)
{
}


Yes, precisely.


+}
+#endif
+
  int amdgpu_do_asic_reset(struct list_head *device_list_handle,
   struct amdgpu_reset_context *reset_context)
  {
@@ -4817,6 +4868,14 @@ int amdgpu_do_asic_reset(struct list_head 
*device_list_handle,

  goto out;
    vram_lost = 
amdgpu_device_check_vram_lost(tmp_adev);

+#ifdef CONFIG_DEV_COREDUMP
+    tmp_adev->reset_context_vram_lost = vram_lost;
+    tmp_adev->reset_context_task_info.pid = 0;

why is the PID hardcoded to 0 ?
in case of reset context reset_context->job->vm  is null (possibility 
that reset can be non VM related).
If we don't set tmp_adev->reset_context_task_info.pid = 0, it will 
show previous reset valid PID.




But when the VM is not NULL, are we updating this PID somewhere ? I 
did not see that happening in this series.

This is the only place where PID get updated.
For example sequence of operation like:
1st reset:
-valid VM and tmp_adev->reset_context_task_info.pid is set to some valid 
PID

2nd reset:
-invalid VM
-tmp_adev context will remain same (adev context will be same after 
su

Re: [PATCH v1 1/2] drm/amdgpu: save the reset dump register value for devcoredump

[Sharma, Shashank]

On 5/24/2022 8:12 AM, Somalapuram, Amaranath wrote:


On 5/20/2022 7:36 PM, Sharma, Shashank wrote:

Hey Amar,

On 5/20/2022 3:49 PM, Somalapuram Amaranath wrote:
Allocate memory for register value and use the same values for 
devcoredump.

Remove dump_stack reset register dumps.

Signed-off-by: Somalapuram Amaranath 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 9 -
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  | 7 +++
  3 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu.h

index 76df583663c7..c79d9992b113 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1042,6 +1042,7 @@ struct amdgpu_device {
    /* reset dump register */
  uint32_t    *reset_dump_reg_list;
+    uint32_t    *reset_dump_reg_value;
  int num_regs;
    struct amdgpu_reset_domain    *reset_domain;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c

index eedb12f6b8a3..942fdbd316f4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
@@ -1683,7 +1683,7 @@ static ssize_t 
amdgpu_reset_dump_register_list_write(struct file *f,

  {
  struct amdgpu_device *adev = (struct amdgpu_device 
*)file_inode(f)->i_private;

  char reg_offset[11];
-    uint32_t *new, *tmp = NULL;
+    uint32_t *new, *tmp = NULL, *tmp_value = NULL;
  int ret, i = 0, len = 0;
    do {
@@ -1709,17 +1709,24 @@ static ssize_t 
amdgpu_reset_dump_register_list_write(struct file *f,

  i++;
  } while (len < size);
  +    new = krealloc_array(tmp_value, i, sizeof(uint32_t), GFP_KERNEL);


tmp_value is initialized to NULL, which means krealloc_array() will 
behave like kmalloc_array(), is there any particular reason we are 
adding this variable at all just to use krealloc_array(), and why not 
use kmalloc_array() directly ?


I thought of using kmalloc_array() (got little confused on next write 
cycle), I agree to use kmalloc_array().


Regards,
S.Amarnath



+    if (!new) {
+    ret = -ENOMEM;
+    goto error_free;
+    }
  ret = down_write_killable(>reset_domain->sem);
  if (ret)
  goto error_free;
    swap(adev->reset_dump_reg_list, tmp);
+    swap(adev->reset_dump_reg_value, new);
  adev->num_regs = i;
  up_write(>reset_domain->sem);
  ret = size;
    error_free:
  kfree(tmp);
+    kfree(new);
  return ret;
  }
  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

index 4daa0e893965..963c897a76e6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -4720,15 +4720,14 @@ int amdgpu_device_pre_asic_reset(struct 
amdgpu_device *adev,

    static int amdgpu_reset_reg_dumps(struct amdgpu_device *adev)
  {
-    uint32_t reg_value;
  int i;
    lockdep_assert_held(>reset_domain->sem);
-    dump_stack();
This should be a part of different patch, where you can give some 
background on why are we removing this.


You missed this comment.
- Shashank


    for (i = 0; i < adev->num_regs; i++) {
-    reg_value = RREG32(adev->reset_dump_reg_list[i]);
- trace_amdgpu_reset_reg_dumps(adev->reset_dump_reg_list[i], reg_value);
+    adev->reset_dump_reg_value[i] = 
RREG32(adev->reset_dump_reg_list[i]);

+ trace_amdgpu_reset_reg_dumps(adev->reset_dump_reg_list[i],
+    adev->reset_dump_reg_value[i]);
  }
    return 0;


- Shashank

Re: [PATCH v1 2/2] drm/amdgpu: adding device coredump support

[Sharma, Shashank]

On 5/24/2022 8:42 AM, Somalapuram, Amaranath wrote:


On 5/20/2022 7:52 PM, Sharma, Shashank wrote:



On 5/20/2022 3:49 PM, Somalapuram Amaranath wrote:

Added device coredump information:
- Kernel version
- Module
- Time
- VRAM status
- Guilty process name and PID
- GPU register dumps

Signed-off-by: Somalapuram Amaranath 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h    |  3 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 59 ++
  2 files changed, 62 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu.h

index c79d9992b113..f28d9c563f74 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1044,6 +1044,9 @@ struct amdgpu_device {
  uint32_t    *reset_dump_reg_list;
  uint32_t    *reset_dump_reg_value;
  int num_regs;
+    struct amdgpu_task_info reset_context_task_info;
+    bool    reset_context_vram_lost;


How about drop the 'context' from name and just reset_task_info and 
reset_vram_lost ?

OK.



+    struct timespec64 reset_time;
    struct amdgpu_reset_domain    *reset_domain;
  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

index 963c897a76e6..f9b710e741a7 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -32,6 +32,8 @@
  #include 
  #include 
  #include 
+#include 
+#include 
    #include 
  #include 
@@ -4733,6 +4735,55 @@ static int amdgpu_reset_reg_dumps(struct 
amdgpu_device *adev)

  return 0;
  }
  +#ifdef CONFIG_DEV_COREDUMP
+static ssize_t amdgpu_devcoredump_read(char *buffer, loff_t offset,
+    size_t count, void *data, size_t datalen)
+{
+    struct drm_printer p;
+    struct amdgpu_device *adev = data;
+    struct drm_print_iterator iter;
+    int i;
+


A NULL check for 'buffer' here could prevent a segfault later.


Agreed.

+    iter.data = buffer;
+    iter.offset = 0;
+    iter.start = offset;
+    iter.remain = count;
+
+    p = drm_coredump_printer();
+
+    drm_printf(, " AMDGPU Device Coredump \n");
+    drm_printf(, "kernel: " UTS_RELEASE "\n");
+    drm_printf(, "module: " KBUILD_MODNAME "\n");
+    drm_printf(, "time: %lld.%09ld\n", adev->reset_time.tv_sec, 
adev->reset_time.tv_nsec);

+    if (adev->reset_context_task_info.pid)
+    drm_printf(, "process_name: %s PID: %d\n",
+ adev->reset_context_task_info.process_name,
+ adev->reset_context_task_info.pid);

Please fix the alignment of print variables.


I will cross check this.

+
+    if (adev->reset_context_vram_lost)
+    drm_printf(, "VRAM is lost due to GPU reset!\n");
+    if (adev->num_regs) {
+    drm_printf(, "AMDGPU register dumps:\nOffset: Value:\n");
+
+    for (i = 0; i < adev->num_regs; i++)
+    drm_printf(, "0x%08x: 0x%08x\n",
+    adev->reset_dump_reg_list[i],
+    adev->reset_dump_reg_value[i]);
+    }
+
+    return count - iter.remain;
+}
+
+static void amdgpu_reset_capture_coredumpm(struct amdgpu_device *adev)
+{
+    struct drm_device *dev = adev_to_drm(adev);
+
+    ktime_get_ts64(>reset_time);
+    dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_KERNEL,
+    amdgpu_devcoredump_read, NULL);
instead of registering NULL as free function, I would prefer you to 
have a dummy no_op free function registered, which we can consume if 
something changes.

you mean something like this (function without any code):
staticvoidamdgpu_devcoredump_free(void*data)
{
}


Yes, precisely.


+}
+#endif
+
  int amdgpu_do_asic_reset(struct list_head *device_list_handle,
   struct amdgpu_reset_context *reset_context)
  {
@@ -4817,6 +4868,14 @@ int amdgpu_do_asic_reset(struct list_head 
*device_list_handle,

  goto out;
    vram_lost = amdgpu_device_check_vram_lost(tmp_adev);
+#ifdef CONFIG_DEV_COREDUMP
+    tmp_adev->reset_context_vram_lost = vram_lost;
+    tmp_adev->reset_context_task_info.pid = 0;

why is the PID hardcoded to 0 ?
in case of reset context reset_context->job->vm  is null (possibility 
that reset can be non VM related).
If we don't set tmp_adev->reset_context_task_info.pid = 0, it will show 
previous reset valid PID.




But when the VM is not NULL, are we updating this PID somewhere ? I did 
not see that happening in this series.


- Shashank



Regards,
S.Amarnath

+    if (reset_context->job && reset_context->job->vm)
+    tmp_adev->reset_context_task_info =
+ reset_context->job->vm->task_info;
+    amdgpu_reset_capture_coredumpm(tmp_adev);
+#endif
  if (vram_lost) {
  DRM_INFO("VRAM is lost due to GPU reset!\n");


- Shashank
 amdgpu_inc_vram_lost(tmp_adev);

Re: [PATCH v1 2/2] drm/amdgpu: adding device coredump support

[Sharma, Shashank]

On 5/20/2022 3:49 PM, Somalapuram Amaranath wrote:

Added device coredump information:
- Kernel version
- Module
- Time
- VRAM status
- Guilty process name and PID
- GPU register dumps

Signed-off-by: Somalapuram Amaranath 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h|  3 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 59 ++
  2 files changed, 62 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index c79d9992b113..f28d9c563f74 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1044,6 +1044,9 @@ struct amdgpu_device {
uint32_t*reset_dump_reg_list;
uint32_t*reset_dump_reg_value;
int num_regs;
+   struct amdgpu_task_info reset_context_task_info;
+   boolreset_context_vram_lost;


How about drop the 'context' from name and just reset_task_info and 
reset_vram_lost ?



+   struct timespec64   reset_time;
  
  	struct amdgpu_reset_domain	*reset_domain;
  
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

index 963c897a76e6..f9b710e741a7 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -32,6 +32,8 @@
  #include 
  #include 
  #include 
+#include 
+#include 
  
  #include 

  #include 
@@ -4733,6 +4735,55 @@ static int amdgpu_reset_reg_dumps(struct amdgpu_device 
*adev)
return 0;
  }
  
+#ifdef CONFIG_DEV_COREDUMP

+static ssize_t amdgpu_devcoredump_read(char *buffer, loff_t offset,
+   size_t count, void *data, size_t datalen)
+{
+   struct drm_printer p;
+   struct amdgpu_device *adev = data;
+   struct drm_print_iterator iter;
+   int i;
+


A NULL check for 'buffer' here could prevent a segfault later.


+   iter.data = buffer;
+   iter.offset = 0;
+   iter.start = offset;
+   iter.remain = count;
+
+   p = drm_coredump_printer();
+
+   drm_printf(, " AMDGPU Device Coredump \n");
+   drm_printf(, "kernel: " UTS_RELEASE "\n");
+   drm_printf(, "module: " KBUILD_MODNAME "\n");
+   drm_printf(, "time: %lld.%09ld\n", adev->reset_time.tv_sec, 
adev->reset_time.tv_nsec);
+   if (adev->reset_context_task_info.pid)
+   drm_printf(, "process_name: %s PID: %d\n",
+   
adev->reset_context_task_info.process_name,
+   
adev->reset_context_task_info.pid);

Please fix the alignment of print variables.


+
+   if (adev->reset_context_vram_lost)
+   drm_printf(, "VRAM is lost due to GPU reset!\n");
+   if (adev->num_regs) {
+   drm_printf(, "AMDGPU register dumps:\nOffset: Value:\n");
+
+   for (i = 0; i < adev->num_regs; i++)
+   drm_printf(, "0x%08x: 0x%08x\n",
+   adev->reset_dump_reg_list[i],
+   adev->reset_dump_reg_value[i]);
+   }
+
+   return count - iter.remain;
+}
+
+static void amdgpu_reset_capture_coredumpm(struct amdgpu_device *adev)
+{
+   struct drm_device *dev = adev_to_drm(adev);
+
+   ktime_get_ts64(>reset_time);
+   dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_KERNEL,
+   amdgpu_devcoredump_read, NULL);
instead of registering NULL as free function, I would prefer you to have 
a dummy no_op free function registered, which we can consume if 
something changes.

+}
+#endif
+
  int amdgpu_do_asic_reset(struct list_head *device_list_handle,
 struct amdgpu_reset_context *reset_context)
  {
@@ -4817,6 +4868,14 @@ int amdgpu_do_asic_reset(struct list_head 
*device_list_handle,
goto out;
  
  vram_lost = amdgpu_device_check_vram_lost(tmp_adev);

+#ifdef CONFIG_DEV_COREDUMP
+   tmp_adev->reset_context_vram_lost = vram_lost;
+   tmp_adev->reset_context_task_info.pid = 0;

why is the PID hardcoded to 0 ?

+   if (reset_context->job && 
reset_context->job->vm)
+   tmp_adev->reset_context_task_info =
+   
reset_context->job->vm->task_info;
+   amdgpu_reset_capture_coredumpm(tmp_adev);
+#endif
if (vram_lost) {
DRM_INFO("VRAM is lost due to GPU 
reset!\n");
 

- Shashank
amdgpu_inc_vram_lost(tmp_adev);

Re: [PATCH v1 1/2] drm/amdgpu: save the reset dump register value for devcoredump

[Sharma, Shashank]

Hey Amar,

On 5/20/2022 3:49 PM, Somalapuram Amaranath wrote:

Allocate memory for register value and use the same values for devcoredump.
Remove dump_stack reset register dumps.

Signed-off-by: Somalapuram Amaranath 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 9 -
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  | 7 +++
  3 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 76df583663c7..c79d9992b113 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1042,6 +1042,7 @@ struct amdgpu_device {
  
  	/* reset dump register */

uint32_t*reset_dump_reg_list;
+   uint32_t*reset_dump_reg_value;
int num_regs;
  
  	struct amdgpu_reset_domain	*reset_domain;

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index eedb12f6b8a3..942fdbd316f4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
@@ -1683,7 +1683,7 @@ static ssize_t 
amdgpu_reset_dump_register_list_write(struct file *f,
  {
struct amdgpu_device *adev = (struct amdgpu_device 
*)file_inode(f)->i_private;
char reg_offset[11];
-   uint32_t *new, *tmp = NULL;
+   uint32_t *new, *tmp = NULL, *tmp_value = NULL;
int ret, i = 0, len = 0;
  
  	do {

@@ -1709,17 +1709,24 @@ static ssize_t 
amdgpu_reset_dump_register_list_write(struct file *f,
i++;
} while (len < size);
  
+	new = krealloc_array(tmp_value, i, sizeof(uint32_t), GFP_KERNEL);


tmp_value is initialized to NULL, which means krealloc_array() will 
behave like kmalloc_array(), is there any particular reason we are 
adding this variable at all just to use krealloc_array(), and why not 
use kmalloc_array() directly ?



+   if (!new) {
+   ret = -ENOMEM;
+   goto error_free;
+   }
ret = down_write_killable(>reset_domain->sem);
if (ret)
goto error_free;
  
  	swap(adev->reset_dump_reg_list, tmp);

+   swap(adev->reset_dump_reg_value, new);
adev->num_regs = i;
up_write(>reset_domain->sem);
ret = size;
  
  error_free:

kfree(tmp);
+   kfree(new);
return ret;
  }
  
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

index 4daa0e893965..963c897a76e6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -4720,15 +4720,14 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device 
*adev,
  
  static int amdgpu_reset_reg_dumps(struct amdgpu_device *adev)

  {
-   uint32_t reg_value;
int i;
  
  	lockdep_assert_held(>reset_domain->sem);

-   dump_stack();
This should be a part of different patch, where you can give some 
background on why are we removing this.
  
  	for (i = 0; i < adev->num_regs; i++) {

-   reg_value = RREG32(adev->reset_dump_reg_list[i]);
-   trace_amdgpu_reset_reg_dumps(adev->reset_dump_reg_list[i], 
reg_value);
+   adev->reset_dump_reg_value[i] = 
RREG32(adev->reset_dump_reg_list[i]);
+   trace_amdgpu_reset_reg_dumps(adev->reset_dump_reg_list[i],
+   adev->reset_dump_reg_value[i]);
}
  
  	return 0;


- Shashank

Re: [PATCH 1/2] drm/amdgpu: Convert to common fdinfo format v5

[Sharma, Shashank]

Please feel free to use:
Reviewed-by: Shashank Sharma 

On 5/17/2022 12:36 PM, Christian König wrote:

Convert fdinfo format to one documented in drm-usage-stats.rst.

It turned out that the existing implementation was actually completely
nonsense. The calculated percentages indeed represented the usage of the
engine, but with varying time slices.

So 10% usage for application A could mean something completely different
than 10% usage for application B.

Completely nuke that and just use the now standardized nanosecond
interface.

v2: drop the documentation change for now, nuke percentage calculation
v3: only account for each hw_ip, move the time_spend to the ctx mgr.
v4: move general ctx changes into separate patch, rework the fdinfo to
 ctx_mgr interface so that all usages are calculated at once, drop
 some unecessary and dangerous refcount dance.
v5: add one more comment how we calculate the time spend

Signed-off-by: Tvrtko Ursulin 
Signed-off-by: Christian König 
Cc: Daniel Vetter 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c| 177 +++--
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h|  12 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_fdinfo.c |  57 +++
  3 files changed, 125 insertions(+), 121 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
index a5fef5213fc5..a1d5275a2e78 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
@@ -162,17 +162,50 @@ static unsigned int amdgpu_ctx_get_hw_prio(struct 
amdgpu_ctx *ctx, u32 hw_ip)
return hw_prio;
  }
  
+/* Calculate the time spend on the hw */

+static ktime_t amdgpu_ctx_fence_time(struct dma_fence *fence)
+{
+   struct drm_sched_fence *s_fence;
+
+   if (!fence)
+   return ns_to_ktime(0);
+
+   /* When the fence is not even scheduled it can't have spend time */
+   s_fence = to_drm_sched_fence(fence);
+   if (!test_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, _fence->scheduled.flags))
+   return ns_to_ktime(0);
+
+   /* When it is still running account how much already spend */
+   if (!test_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, _fence->finished.flags))
+   return ktime_sub(ktime_get(), s_fence->scheduled.timestamp);
+
+   return ktime_sub(s_fence->finished.timestamp,
+s_fence->scheduled.timestamp);
+}
+
+static ktime_t amdgpu_ctx_entity_time(struct amdgpu_ctx *ctx,
+ struct amdgpu_ctx_entity *centity)
+{
+   ktime_t res = ns_to_ktime(0);
+   uint32_t i;
+
+   spin_lock(>ring_lock);
+   for (i = 0; i < amdgpu_sched_jobs; i++) {
+   res = ktime_add(res, amdgpu_ctx_fence_time(centity->fences[i]));
+   }
+   spin_unlock(>ring_lock);
+   return res;
+}
  
  static int amdgpu_ctx_init_entity(struct amdgpu_ctx *ctx, u32 hw_ip,

  const u32 ring)
  {
+   struct drm_gpu_scheduler **scheds = NULL, *sched = NULL;
struct amdgpu_device *adev = ctx->mgr->adev;
struct amdgpu_ctx_entity *entity;
-   struct drm_gpu_scheduler **scheds = NULL, *sched = NULL;
-   unsigned num_scheds = 0;
-   int32_t ctx_prio;
-   unsigned int hw_prio;
enum drm_sched_priority drm_prio;
+   unsigned int hw_prio, num_scheds;
+   int32_t ctx_prio;
int r;
  
  	entity = kzalloc(struct_size(entity, fences, amdgpu_sched_jobs),

@@ -182,6 +215,7 @@ static int amdgpu_ctx_init_entity(struct amdgpu_ctx *ctx, 
u32 hw_ip,
  
  	ctx_prio = (ctx->override_priority == AMDGPU_CTX_PRIORITY_UNSET) ?

ctx->init_priority : ctx->override_priority;
+   entity->hw_ip = hw_ip;
entity->sequence = 1;
hw_prio = amdgpu_ctx_get_hw_prio(ctx, hw_ip);
drm_prio = amdgpu_ctx_to_drm_sched_prio(ctx_prio);
@@ -220,6 +254,23 @@ static int amdgpu_ctx_init_entity(struct amdgpu_ctx *ctx, 
u32 hw_ip,
return r;
  }
  
+static ktime_t amdgpu_ctx_fini_entity(struct amdgpu_ctx_entity *entity)

+{
+   ktime_t res = ns_to_ktime(0);
+   int i;
+
+   if (!entity)
+   return res;
+
+   for (i = 0; i < amdgpu_sched_jobs; ++i) {
+   res = ktime_add(res, amdgpu_ctx_fence_time(entity->fences[i]));
+   dma_fence_put(entity->fences[i]);
+   }
+
+   kfree(entity);
+   return res;
+}
+
  static int amdgpu_ctx_init(struct amdgpu_ctx_mgr *mgr, int32_t priority,
   struct drm_file *filp, struct amdgpu_ctx *ctx)
  {
@@ -245,20 +296,6 @@ static int amdgpu_ctx_init(struct amdgpu_ctx_mgr *mgr, 
int32_t priority,
return 0;
  }
  
-static void amdgpu_ctx_fini_entity(struct amdgpu_ctx_entity *entity)

-{
-
-   int i;
-
-   if (!entity)
-   return;
-
-   for (i = 0; i < amdgpu_sched_jobs; ++i)
-   dma_fence_put(entity->fences[i]);
-
-   kfree(entity);
-}
-
  static int 

Re: [PATCH 2/2] drm/amdgpu: add drm-client-id to fdinfo

[Sharma, Shashank]

On 5/11/2022 2:02 PM, Christian König wrote:

This is enough to get gputop working :)

Signed-off-by: Christian König 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_fdinfo.c | 7 +++
  1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fdinfo.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_fdinfo.c
index 52c2b90925a0..780a48259682 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fdinfo.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fdinfo.c
@@ -55,17 +55,15 @@ static const char *amdgpu_ip_name[AMDGPU_HW_IP_NUM] = {
  
  void amdgpu_show_fdinfo(struct seq_file *m, struct file *f)

  {
-   struct amdgpu_fpriv *fpriv;
uint64_t vram_mem = 0, gtt_mem = 0, cpu_mem = 0;
struct drm_file *file = f->private_data;
struct amdgpu_device *adev = drm_to_adev(file->minor->dev);
+   struct amdgpu_fpriv *fpriv = file->driver_priv;
+   struct amdgpu_vm *vm = >vm;
uint32_t bus, dev, fn, domain, hw_ip;
struct amdgpu_bo *root;
int ret;
  
-	ret = amdgpu_file_to_fpriv(f, );

-   if (ret)
-   return;
bus = adev->pdev->bus->number;
domain = pci_domain_nr(adev->pdev->bus);
dev = PCI_SLOT(adev->pdev->devfn);
@@ -93,6 +91,7 @@ void amdgpu_show_fdinfo(struct seq_file *m, struct file *f)
seq_printf(m, "drm-driver:\t%s\n", file->minor->dev->driver->name);
seq_printf(m, "drm-pdev:\t%04x:%02x:%02x.%d\npasid:\t%u\n", domain, bus,
dev, fn, fpriv->vm.pasid);
+   seq_printf(m, "drm-client-id:\t%Lu\n", vm->immediate.fence_context);
seq_printf(m, "drm-memory-vram:\t%llu KiB\n", vram_mem/1024UL);
seq_printf(m, "drm-memory-gtt:\t%llu KiB\n", gtt_mem/1024UL);
seq_printf(m, "drm-memory-cpu:\t%llu KiB\n", cpu_mem/1024UL);


Lgtm, please feel free to use
Reviewed-by: Shashank Sharma 

Re: [PATCH 1/2] drm/amdgpu: Convert to common fdinfo format v3

[Sharma, Shashank]

Hey Christian,

On 5/11/2022 2:02 PM, Christian König wrote:

Convert fdinfo format to one documented in drm-usage-stats.rst.

It turned out that the existing implementation was actually completely
nonsense. The calculated percentages indeed represented the usage of the
engine, but with varying time slices.

So 10% usage for application A could mean something completely different
than 10% usage for application B.

Completely nuke that and just use the now standardized nanosecond
interface.

v2: drop the documentation change for now, nuke percentage calculation
v3: only account for each hw_ip, move the time_spend to the ctx mgr.

Signed-off-by: Tvrtko Ursulin 
Signed-off-by: Christian König 
Cc: David M Nieto 
Cc: Daniel Vetter 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c| 234 ++---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h|  23 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_fdinfo.c |  41 ++--
  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c|   2 +-
  5 files changed, 153 insertions(+), 149 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
index 01853431249d..43b97ad3c6be 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
@@ -1239,7 +1239,7 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,
  
  	p->fence = dma_fence_get(>base.s_fence->finished);
  
-	amdgpu_ctx_add_fence(p->ctx, entity, p->fence, );

+   seq = amdgpu_ctx_add_fence(>ctx_mgr, p->ctx, entity, p->fence);
amdgpu_cs_post_dependencies(p);
  
  	if ((job->preamble_status & AMDGPU_PREAMBLE_IB_PRESENT) &&

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
index 8f0e6d93bb9c..5a0d67cc3d75 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
@@ -135,9 +135,9 @@ static enum amdgpu_ring_priority_level 
amdgpu_ctx_sched_prio_to_ring_prio(int32_
  
  static unsigned int amdgpu_ctx_get_hw_prio(struct amdgpu_ctx *ctx, u32 hw_ip)

  {
-   struct amdgpu_device *adev = ctx->adev;
-   int32_t ctx_prio;
+   struct amdgpu_device *adev = ctx->mgr->adev;
unsigned int hw_prio;
+   int32_t ctx_prio;
  
  	ctx_prio = (ctx->override_priority == AMDGPU_CTX_PRIORITY_UNSET) ?

ctx->init_priority : ctx->override_priority;
@@ -162,17 +162,49 @@ static unsigned int amdgpu_ctx_get_hw_prio(struct 
amdgpu_ctx *ctx, u32 hw_ip)
return hw_prio;
  }
  
+/* Calculate the time spend on the hw */

+static ktime_t amdgpu_ctx_fence_time(struct dma_fence *fence)
+{
+   struct drm_sched_fence *s_fence;
+
+   if (!fence)
+   return ns_to_ktime(0);
+
+   /* When the fence is not even scheduled it can't have spend time */
+   s_fence = to_drm_sched_fence(fence);
+   if (!test_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, _fence->scheduled.flags))
+   return ns_to_ktime(0);
+
+   if (!test_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, _fence->finished.flags))
+   return ktime_sub(ktime_get(), s_fence->scheduled.timestamp);
shouldn't this be s_fence->finished.timestamp instead of 
s_fence->scheduled.timestamp ?

+
+   return ktime_sub(s_fence->finished.timestamp,
+s_fence->scheduled.timestamp);
+}
+
+static ktime_t amdgpu_ctx_entity_time(struct amdgpu_ctx *ctx,
+ struct amdgpu_ctx_entity *centity)
+{
+   ktime_t res = ns_to_ktime(0);
+   uint32_t i;
+
+   spin_lock(>ring_lock);
+   for (i = 0; i < amdgpu_sched_jobs; i++) {
+   res = ktime_add(res, amdgpu_ctx_fence_time(centity->fences[i]));
+   }
+   spin_unlock(>ring_lock);
+   return res;
+}
  
  static int amdgpu_ctx_init_entity(struct amdgpu_ctx *ctx, u32 hw_ip,

  const u32 ring)
  {
-   struct amdgpu_device *adev = ctx->adev;
-   struct amdgpu_ctx_entity *entity;
struct drm_gpu_scheduler **scheds = NULL, *sched = NULL;
-   unsigned num_scheds = 0;
-   int32_t ctx_prio;
-   unsigned int hw_prio;
+   struct amdgpu_device *adev = ctx->mgr->adev;
+   struct amdgpu_ctx_entity *entity;
enum drm_sched_priority drm_prio;
+   unsigned int hw_prio, num_scheds;
+   int32_t ctx_prio;
int r;
  
  	entity = kzalloc(struct_size(entity, fences, amdgpu_sched_jobs),

@@ -182,6 +214,7 @@ static int amdgpu_ctx_init_entity(struct amdgpu_ctx *ctx, 
u32 hw_ip,
  
  	ctx_prio = (ctx->override_priority == AMDGPU_CTX_PRIORITY_UNSET) ?

ctx->init_priority : ctx->override_priority;
+   entity->hw_ip = hw_ip;
entity->sequence = 1;
hw_prio = amdgpu_ctx_get_hw_prio(ctx, hw_ip);
drm_prio = amdgpu_ctx_to_drm_sched_prio(ctx_prio);
@@ -220,11 +253,29 @@ static int amdgpu_ctx_init_entity(struct amdgpu_ctx *ctx, 
u32 hw_ip,
return r;
  }
  
-static 

Re: [PATCH v2 1/2] drm: Add GPU reset sysfs event

[Sharma, Shashank]

On 3/16/2022 10:50 PM, Rob Clark wrote:

On Tue, Mar 8, 2022 at 11:40 PM Shashank Sharma
 wrote:


From: Shashank Sharma 

This patch adds a new sysfs event, which will indicate
the userland about a GPU reset, and can also provide
some information like:
- process ID of the process involved with the GPU reset
- process name of the involved process
- the GPU status info (using flags)

This patch also introduces the first flag of the flags
bitmap, which can be appended as and when required.

V2: Addressed review comments from Christian and Amar
- move the reset information structure to DRM layer
- drop _ctx from struct name
- make pid 32 bit(than 64)
- set flag when VRAM invalid (than valid)
- add process name as well (Amar)

Cc: Alexandar Deucher 
Cc: Christian Koenig 
Cc: Amaranath Somalapuram 
Signed-off-by: Shashank Sharma 
---
  drivers/gpu/drm/drm_sysfs.c | 31 +++
  include/drm/drm_sysfs.h | 10 ++
  2 files changed, 41 insertions(+)

diff --git a/drivers/gpu/drm/drm_sysfs.c b/drivers/gpu/drm/drm_sysfs.c
index 430e00b16eec..840994810910 100644
--- a/drivers/gpu/drm/drm_sysfs.c
+++ b/drivers/gpu/drm/drm_sysfs.c
@@ -409,6 +409,37 @@ void drm_sysfs_hotplug_event(struct drm_device *dev)
  }
  EXPORT_SYMBOL(drm_sysfs_hotplug_event);

+/**
+ * drm_sysfs_reset_event - generate a DRM uevent to indicate GPU reset
+ * @dev: DRM device
+ * @reset_info: The contextual information about the reset (like PID, flags)
+ *
+ * Send a uevent for the DRM device specified by @dev. This informs
+ * user that a GPU reset has occurred, so that an interested client
+ * can take any recovery or profiling measure.
+ */
+void drm_sysfs_reset_event(struct drm_device *dev, struct drm_reset_event 
*reset_info)
+{
+   unsigned char pid_str[13];
+   unsigned char flags_str[15];
+   unsigned char pname_str[TASK_COMM_LEN + 6];
+   unsigned char reset_str[] = "RESET=1";
+   char *envp[] = { reset_str, pid_str, pname_str, flags_str, NULL };
+
+   if (!reset_info) {
+   DRM_WARN("No reset info, not sending the event\n");
+   return;
+   }
+
+   DRM_DEBUG("generating reset event\n");
+
+   snprintf(pid_str, ARRAY_SIZE(pid_str), "PID=%u", reset_info->pid);
+   snprintf(pname_str, ARRAY_SIZE(pname_str), "NAME=%s", 
reset_info->pname);
+   snprintf(flags_str, ARRAY_SIZE(flags_str), "FLAGS=%u", 
reset_info->flags);
+   kobject_uevent_env(>primary->kdev->kobj, KOBJ_CHANGE, envp);
+}
+EXPORT_SYMBOL(drm_sysfs_reset_event);
+
  /**
   * drm_sysfs_connector_hotplug_event - generate a DRM uevent for any connector
   * change
diff --git a/include/drm/drm_sysfs.h b/include/drm/drm_sysfs.h
index 6273cac44e47..5ba11c760619 100644
--- a/include/drm/drm_sysfs.h
+++ b/include/drm/drm_sysfs.h
@@ -1,16 +1,26 @@
  /* SPDX-License-Identifier: GPL-2.0 */
  #ifndef _DRM_SYSFS_H_
  #define _DRM_SYSFS_H_
+#include 
+
+#define DRM_GPU_RESET_FLAG_VRAM_INVALID (1 << 0)

  struct drm_device;
  struct device;
  struct drm_connector;
  struct drm_property;

+struct drm_reset_event {
+   uint32_t pid;


One side note, unrelated to devcoredump vs this..

AFAIU you probably want to be passing around a `struct pid *`, and
then somehow use pid_vnr() in the context of the process reading the
event to get the numeric pid.  Otherwise things will not do what you
expect if the process triggering the crash is in a different pid
namespace from the compositor.



I am not sure if it is a good idea to add the pid extraction complexity 
in here, it is left upto the driver to extract this information and pass 
it to the work queue. In case of AMDGPU, its extracted from GPU VM. It 
would be then more flexible for the drivers as well.


- Shashank


BR,
-R


+   uint32_t flags;
+   char pname[TASK_COMM_LEN];
+};
+
  int drm_class_device_register(struct device *dev);
  void drm_class_device_unregister(struct device *dev);

  void drm_sysfs_hotplug_event(struct drm_device *dev);
+void drm_sysfs_reset_event(struct drm_device *dev, struct drm_reset_event 
*reset_info);
  void drm_sysfs_connector_hotplug_event(struct drm_connector *connector);
  void drm_sysfs_connector_status_event(struct drm_connector *connector,
   struct drm_property *property);
--
2.32.0

Re: [PATCH v2 1/2] drm: Add GPU reset sysfs event

[Sharma, Shashank]

On 3/10/2022 8:56 PM, Rob Clark wrote:

On Thu, Mar 10, 2022 at 11:44 AM Sharma, Shashank
 wrote:




On 3/10/2022 8:35 PM, Rob Clark wrote:

On Thu, Mar 10, 2022 at 11:14 AM Sharma, Shashank
 wrote:




On 3/10/2022 7:33 PM, Abhinav Kumar wrote:



On 3/10/2022 9:40 AM, Rob Clark wrote:

On Thu, Mar 10, 2022 at 9:19 AM Sharma, Shashank
 wrote:




On 3/10/2022 6:10 PM, Rob Clark wrote:

On Thu, Mar 10, 2022 at 8:21 AM Sharma, Shashank
 wrote:




On 3/10/2022 4:24 PM, Rob Clark wrote:

On Thu, Mar 10, 2022 at 1:55 AM Christian König
 wrote:




Am 09.03.22 um 19:12 schrieb Rob Clark:

On Tue, Mar 8, 2022 at 11:40 PM Shashank Sharma
 wrote:

From: Shashank Sharma 

This patch adds a new sysfs event, which will indicate
the userland about a GPU reset, and can also provide
some information like:
- process ID of the process involved with the GPU reset
- process name of the involved process
- the GPU status info (using flags)

This patch also introduces the first flag of the flags
bitmap, which can be appended as and when required.

Why invent something new, rather than using the already existing
devcoredump?


Yeah, that's a really valid question.


I don't think we need (or should encourage/allow) something drm
specific when there is already an existing solution used by both
drm
and non-drm drivers.  Userspace should not have to learn to support
yet another mechanism to do the same thing.


Question is how is userspace notified about new available core
dumps?


I haven't looked into it too closely, as the CrOS userspace
crash-reporter already had support for devcoredump, so it "just
worked" out of the box[1].  I believe a udev event is what triggers
the crash-reporter to go read the devcore dump out of sysfs.


I had a quick look at the devcoredump code, and it doesn't look like
that is sending an event to the user, so we still need an event to
indicate a GPU reset.


There definitely is an event to userspace, I suspect somewhere down
the device_add() path?



Let me check that out as well, hope that is not due to a driver-private
event for GPU reset, coz I think I have seen some of those in a few DRM
drivers.



Definitely no driver private event for drm/msm .. I haven't dug
through it all but this is the collector for devcoredump, triggered
somehow via udev.  Most likely from event triggered by device_add()

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fchromium.googlesource.com%2Fchromiumos%2Fplatform2%2F%2B%2FHEAD%2Fcrash-reporter%2Fudev_collector.ccdata=04%7C01%7Cshashank.sharma%40amd.com%7C3b5c0e8744234962061d08da02d00248%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637825389694363926%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=mWI1Z%2B8eMJApePc5ajRipbGUG9Cw9wXf2FCw6NQxVaM%3Dreserved=0



Yes, that is correct. the uevent for devcoredump is from device_add()


Yes, I could confirm in the code that device_add() sends a uevent.

kobject_uevent(>kobj, KOBJ_ADD);

I was trying to map the ChromiumOs's udev event rules with the event
being sent from device_add(), what I could see is there is only one udev
rule for any DRM subsystem events in ChromiumOs's 99-crash-reporter.rules:

ACTION=="change", SUBSYSTEM=="drm", KERNEL=="card0", ENV{ERROR}=="1", \
 RUN+="/sbin/crash_reporter
--udev=KERNEL=card0:SUBSYSTEM=drm:ACTION=change"

Can someone confirm that this is the rule which gets triggered when a
devcoredump is generated ? I could not find an ERROR=1 string in the
env[] while sending this event from dev_add();


I think it is actually this rule:

ACTION=="add", SUBSYSTEM=="devcoredump", \
RUN+="/sbin/crash_reporter
--udev=SUBSYSTEM=devcoredump:ACTION=add:KERNEL_NUMBER=%n"

It is something non-drm specific because it supports devcore dumps
from non drm drivers.  I know at least some of the wifi and remoteproc
drivers use it.



Ah, this seems like a problem for me. I understand it will work for a
reset/recovery app well, but if a DRM client (like a compositor), who
wants to listen only to DRM events (like a GPU reset), wouldn't this
create a lot of noise for it ? Like every time any subsystem produces
this coredump, there will be a change in devcoresump subsystem, and the
client will have to parse the core file, and then will have to decide if
it wants to react to this, or ignore.

Wouldn't a GPU reset event, specific to DRM subsystem server better in
such case ?



So, I suppose there are two different use-cases.. for something like
distro which has generic crash telemetry (ie. when users opt in to
automated crash reporting), and in general for debugging gpu crashes,
you want devcoredump, preferably with plenty of information about gpu
state, etc, so you actually have a chance of debugging problems you
can't necessarily reproduce locally.  Note also that mesa CI has some
limited support for collecting devcore dumps if a 

Re: [PATCH v2 1/2] drm: Add GPU reset sysfs event

[Sharma, Shashank]

On 3/10/2022 8:35 PM, Rob Clark wrote:

On Thu, Mar 10, 2022 at 11:14 AM Sharma, Shashank
 wrote:




On 3/10/2022 7:33 PM, Abhinav Kumar wrote:



On 3/10/2022 9:40 AM, Rob Clark wrote:

On Thu, Mar 10, 2022 at 9:19 AM Sharma, Shashank
 wrote:




On 3/10/2022 6:10 PM, Rob Clark wrote:

On Thu, Mar 10, 2022 at 8:21 AM Sharma, Shashank
 wrote:




On 3/10/2022 4:24 PM, Rob Clark wrote:

On Thu, Mar 10, 2022 at 1:55 AM Christian König
 wrote:




Am 09.03.22 um 19:12 schrieb Rob Clark:

On Tue, Mar 8, 2022 at 11:40 PM Shashank Sharma
 wrote:

From: Shashank Sharma 

This patch adds a new sysfs event, which will indicate
the userland about a GPU reset, and can also provide
some information like:
- process ID of the process involved with the GPU reset
- process name of the involved process
- the GPU status info (using flags)

This patch also introduces the first flag of the flags
bitmap, which can be appended as and when required.

Why invent something new, rather than using the already existing
devcoredump?


Yeah, that's a really valid question.


I don't think we need (or should encourage/allow) something drm
specific when there is already an existing solution used by both
drm
and non-drm drivers.  Userspace should not have to learn to support
yet another mechanism to do the same thing.


Question is how is userspace notified about new available core
dumps?


I haven't looked into it too closely, as the CrOS userspace
crash-reporter already had support for devcoredump, so it "just
worked" out of the box[1].  I believe a udev event is what triggers
the crash-reporter to go read the devcore dump out of sysfs.


I had a quick look at the devcoredump code, and it doesn't look like
that is sending an event to the user, so we still need an event to
indicate a GPU reset.


There definitely is an event to userspace, I suspect somewhere down
the device_add() path?



Let me check that out as well, hope that is not due to a driver-private
event for GPU reset, coz I think I have seen some of those in a few DRM
drivers.



Definitely no driver private event for drm/msm .. I haven't dug
through it all but this is the collector for devcoredump, triggered
somehow via udev.  Most likely from event triggered by device_add()

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fchromium.googlesource.com%2Fchromiumos%2Fplatform2%2F%2B%2FHEAD%2Fcrash-reporter%2Fudev_collector.ccdata=04%7C01%7Cshashank.sharma%40amd.com%7Cb4e920f125ae4d7de29708da02cd3112%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637825377562005233%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=M4xHPErex4vn7l3lNPgniiMp%2BKb3SpOHQo2QLAndxDQ%3Dreserved=0



Yes, that is correct. the uevent for devcoredump is from device_add()


Yes, I could confirm in the code that device_add() sends a uevent.

kobject_uevent(>kobj, KOBJ_ADD);

I was trying to map the ChromiumOs's udev event rules with the event
being sent from device_add(), what I could see is there is only one udev
rule for any DRM subsystem events in ChromiumOs's 99-crash-reporter.rules:

ACTION=="change", SUBSYSTEM=="drm", KERNEL=="card0", ENV{ERROR}=="1", \
RUN+="/sbin/crash_reporter
--udev=KERNEL=card0:SUBSYSTEM=drm:ACTION=change"

Can someone confirm that this is the rule which gets triggered when a
devcoredump is generated ? I could not find an ERROR=1 string in the
env[] while sending this event from dev_add();


I think it is actually this rule:

ACTION=="add", SUBSYSTEM=="devcoredump", \
   RUN+="/sbin/crash_reporter
--udev=SUBSYSTEM=devcoredump:ACTION=add:KERNEL_NUMBER=%n"

It is something non-drm specific because it supports devcore dumps
from non drm drivers.  I know at least some of the wifi and remoteproc
drivers use it.



Ah, this seems like a problem for me. I understand it will work for a 
reset/recovery app well, but if a DRM client (like a compositor), who 
wants to listen only to DRM events (like a GPU reset), wouldn't this 
create a lot of noise for it ? Like every time any subsystem produces 
this coredump, there will be a change in devcoresump subsystem, and the 
client will have to parse the core file, and then will have to decide if 
it wants to react to this, or ignore.


Wouldn't a GPU reset event, specific to DRM subsystem server better in 
such case ?


- Shashank


BR,
-R

Re: [PATCH v2 1/2] drm: Add GPU reset sysfs event

[Sharma, Shashank]

On 3/10/2022 7:33 PM, Abhinav Kumar wrote:



On 3/10/2022 9:40 AM, Rob Clark wrote:

On Thu, Mar 10, 2022 at 9:19 AM Sharma, Shashank
 wrote:




On 3/10/2022 6:10 PM, Rob Clark wrote:

On Thu, Mar 10, 2022 at 8:21 AM Sharma, Shashank
 wrote:




On 3/10/2022 4:24 PM, Rob Clark wrote:

On Thu, Mar 10, 2022 at 1:55 AM Christian König
 wrote:




Am 09.03.22 um 19:12 schrieb Rob Clark:

On Tue, Mar 8, 2022 at 11:40 PM Shashank Sharma
 wrote:

From: Shashank Sharma 

This patch adds a new sysfs event, which will indicate
the userland about a GPU reset, and can also provide
some information like:
- process ID of the process involved with the GPU reset
- process name of the involved process
- the GPU status info (using flags)

This patch also introduces the first flag of the flags
bitmap, which can be appended as and when required.
Why invent something new, rather than using the already existing 
devcoredump?


Yeah, that's a really valid question.


I don't think we need (or should encourage/allow) something drm
specific when there is already an existing solution used by both 
drm

and non-drm drivers.  Userspace should not have to learn to support
yet another mechanism to do the same thing.


Question is how is userspace notified about new available core 
dumps?


I haven't looked into it too closely, as the CrOS userspace
crash-reporter already had support for devcoredump, so it "just
worked" out of the box[1].  I believe a udev event is what triggers
the crash-reporter to go read the devcore dump out of sysfs.


I had a quick look at the devcoredump code, and it doesn't look like
that is sending an event to the user, so we still need an event to
indicate a GPU reset.


There definitely is an event to userspace, I suspect somewhere down
the device_add() path?



Let me check that out as well, hope that is not due to a driver-private
event for GPU reset, coz I think I have seen some of those in a few DRM
drivers.



Definitely no driver private event for drm/msm .. I haven't dug
through it all but this is the collector for devcoredump, triggered
somehow via udev.  Most likely from event triggered by device_add()

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fchromium.googlesource.com%2Fchromiumos%2Fplatform2%2F%2B%2FHEAD%2Fcrash-reporter%2Fudev_collector.ccdata=04%7C01%7Cshashank.sharma%40amd.com%7C86146416b717420501fc08da02c4785b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637825340130157925%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=LncI%2F5mIpeG1Avj2YXLmbZ5f1ONUfpf6TzJZH3%2Fs8%2Fw%3Dreserved=0 



Yes, that is correct. the uevent for devcoredump is from device_add()


Yes, I could confirm in the code that device_add() sends a uevent.

kobject_uevent(>kobj, KOBJ_ADD);

I was trying to map the ChromiumOs's udev event rules with the event 
being sent from device_add(), what I could see is there is only one udev 
rule for any DRM subsystem events in ChromiumOs's 99-crash-reporter.rules:


ACTION=="change", SUBSYSTEM=="drm", KERNEL=="card0", ENV{ERROR}=="1", \
  RUN+="/sbin/crash_reporter 
--udev=KERNEL=card0:SUBSYSTEM=drm:ACTION=change"


Can someone confirm that this is the rule which gets triggered when a 
devcoredump is generated ? I could not find an ERROR=1 string in the 
env[] while sending this event from dev_add();


- Shashank

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftorvalds%2Flinux%2Fblob%2Fmaster%2Fdrivers%2Fbase%2Fcore.c%23L3340data=04%7C01%7Cshashank.sharma%40amd.com%7C86146416b717420501fc08da02c4785b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637825340130157925%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=5HyWYZ5ZWYz4mUPWeTW51QFdoY0NlA50Nbj1dAC6os4%3Dreserved=0 





BR,
-R