Re: [PATCH v2 1/6] drm/ci: uprev mesa version

2024-05-23 Thread Vignesh Raman 

Hi Dmitry,

On 23/05/24 15:29, Dmitry Baryshkov wrote:

On Thu, 23 May 2024 at 09:07, Vignesh Raman  wrote:


Hi Dmitry,

On 20/05/24 16:13, Dmitry Baryshkov wrote:

On Fri, May 17, 2024 at 02:54:57PM +0530, Vignesh Raman wrote:

zlib.net is not allowing tarball download anymore and results
in below error in kernel+rootfs_arm32 container build,
urllib.error.HTTPError: HTTP Error 403: Forbidden
urllib.error.HTTPError: HTTP Error 415: Unsupported Media Type

Uprev mesa to latest version which includes a fix for this issue.
https://gitlab.freedesktop.org/mesa/mesa/-/commit/908f444e

Use id_tokens for JWT authentication. Since s3 bucket is migrated to
mesa-rootfs, update the variables accordingly. Also copy helper scripts
to install, so that the ci jobs can use these scripts for logging.

Signed-off-by: Vignesh Raman 
---

v2:
- Uprev to recent version and use id_tokens for JWT authentication

---
   drivers/gpu/drm/ci/build-igt.sh   |  2 +-
   drivers/gpu/drm/ci/build.sh   |  6 +++--
   drivers/gpu/drm/ci/container.yml  | 12 +++--
   drivers/gpu/drm/ci/gitlab-ci.yml  | 44 +--
   drivers/gpu/drm/ci/image-tags.yml |  2 +-
   drivers/gpu/drm/ci/lava-submit.sh |  4 +--
   drivers/gpu/drm/ci/test.yml   |  2 ++
   7 files changed, 44 insertions(+), 28 deletions(-)



[skipped]


diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
index 8bc63912fddb..612c9ede3507 100644
--- a/drivers/gpu/drm/ci/test.yml
+++ b/drivers/gpu/drm/ci/test.yml
@@ -150,6 +150,8 @@ msm:sdm845:
   BM_KERNEL: https://${PIPELINE_ARTIFACTS_BASE}/arm64/cheza-kernel
   GPU_VERSION: sdm845
   RUNNER_TAG: google-freedreno-cheza
+DEVICE_TYPE: sdm845-cheza-r3
+FARM: google


I see that this is the only user of the FARM: tag. Is it correct?


No, we need to add FARM variable for other jobs as well.


Why? Even if we have to, we don't have them now and the change doesn't
seem to be related to the uprev'ing of mesa. So this probably should
go to a separate commit.


I will move them to separate commit.

Regards,
Vignesh






Also we miss DEVICE_TYPE for several other boards. Should we be adding
them?


Yes, device type needs to be added for msm:apq8016, msm:apq8096, virtio_gpu.

I will add this. Thanks.


I'd guess, separate commit too.



Regards,
Vignesh




 script:
   - ./install/bare-metal/cros-servo.sh

--
2.40.1

Re: [PATCH v2 1/6] drm/ci: uprev mesa version

2024-05-23 Thread Dmitry Baryshkov 
On Thu, 23 May 2024 at 09:07, Vignesh Raman  wrote:
>
> Hi Dmitry,
>
> On 20/05/24 16:13, Dmitry Baryshkov wrote:
> > On Fri, May 17, 2024 at 02:54:57PM +0530, Vignesh Raman wrote:
> >> zlib.net is not allowing tarball download anymore and results
> >> in below error in kernel+rootfs_arm32 container build,
> >> urllib.error.HTTPError: HTTP Error 403: Forbidden
> >> urllib.error.HTTPError: HTTP Error 415: Unsupported Media Type
> >>
> >> Uprev mesa to latest version which includes a fix for this issue.
> >> https://gitlab.freedesktop.org/mesa/mesa/-/commit/908f444e
> >>
> >> Use id_tokens for JWT authentication. Since s3 bucket is migrated to
> >> mesa-rootfs, update the variables accordingly. Also copy helper scripts
> >> to install, so that the ci jobs can use these scripts for logging.
> >>
> >> Signed-off-by: Vignesh Raman 
> >> ---
> >>
> >> v2:
> >>- Uprev to recent version and use id_tokens for JWT authentication
> >>
> >> ---
> >>   drivers/gpu/drm/ci/build-igt.sh   |  2 +-
> >>   drivers/gpu/drm/ci/build.sh   |  6 +++--
> >>   drivers/gpu/drm/ci/container.yml  | 12 +++--
> >>   drivers/gpu/drm/ci/gitlab-ci.yml  | 44 +--
> >>   drivers/gpu/drm/ci/image-tags.yml |  2 +-
> >>   drivers/gpu/drm/ci/lava-submit.sh |  4 +--
> >>   drivers/gpu/drm/ci/test.yml   |  2 ++
> >>   7 files changed, 44 insertions(+), 28 deletions(-)
> >>
> >
> > [skipped]
> >
> >> diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
> >> index 8bc63912fddb..612c9ede3507 100644
> >> --- a/drivers/gpu/drm/ci/test.yml
> >> +++ b/drivers/gpu/drm/ci/test.yml
> >> @@ -150,6 +150,8 @@ msm:sdm845:
> >>   BM_KERNEL: https://${PIPELINE_ARTIFACTS_BASE}/arm64/cheza-kernel
> >>   GPU_VERSION: sdm845
> >>   RUNNER_TAG: google-freedreno-cheza
> >> +DEVICE_TYPE: sdm845-cheza-r3
> >> +FARM: google
> >
> > I see that this is the only user of the FARM: tag. Is it correct?
>
> No, we need to add FARM variable for other jobs as well.

Why? Even if we have to, we don't have them now and the change doesn't
seem to be related to the uprev'ing of mesa. So this probably should
go to a separate commit.

>
> > Also we miss DEVICE_TYPE for several other boards. Should we be adding
> > them?
>
> Yes, device type needs to be added for msm:apq8016, msm:apq8096, virtio_gpu.
>
> I will add this. Thanks.

I'd guess, separate commit too.

>
> Regards,
> Vignesh
>
> >
> >> script:
> >>   - ./install/bare-metal/cros-servo.sh
> >>
> >> --
> >> 2.40.1
> >>
> >



-- 
With best wishes
Dmitry

Re: [PATCH v2 1/6] drm/ci: uprev mesa version

2024-05-22 Thread Vignesh Raman 

Hi Dmitry,

On 20/05/24 16:13, Dmitry Baryshkov wrote:

On Fri, May 17, 2024 at 02:54:57PM +0530, Vignesh Raman wrote:

zlib.net is not allowing tarball download anymore and results
in below error in kernel+rootfs_arm32 container build,
urllib.error.HTTPError: HTTP Error 403: Forbidden
urllib.error.HTTPError: HTTP Error 415: Unsupported Media Type

Uprev mesa to latest version which includes a fix for this issue.
https://gitlab.freedesktop.org/mesa/mesa/-/commit/908f444e

Use id_tokens for JWT authentication. Since s3 bucket is migrated to
mesa-rootfs, update the variables accordingly. Also copy helper scripts
to install, so that the ci jobs can use these scripts for logging.

Signed-off-by: Vignesh Raman 
---

v2:
   - Uprev to recent version and use id_tokens for JWT authentication

---
  drivers/gpu/drm/ci/build-igt.sh   |  2 +-
  drivers/gpu/drm/ci/build.sh   |  6 +++--
  drivers/gpu/drm/ci/container.yml  | 12 +++--
  drivers/gpu/drm/ci/gitlab-ci.yml  | 44 +--
  drivers/gpu/drm/ci/image-tags.yml |  2 +-
  drivers/gpu/drm/ci/lava-submit.sh |  4 +--
  drivers/gpu/drm/ci/test.yml   |  2 ++
  7 files changed, 44 insertions(+), 28 deletions(-)



[skipped]


diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
index 8bc63912fddb..612c9ede3507 100644
--- a/drivers/gpu/drm/ci/test.yml
+++ b/drivers/gpu/drm/ci/test.yml
@@ -150,6 +150,8 @@ msm:sdm845:
  BM_KERNEL: https://${PIPELINE_ARTIFACTS_BASE}/arm64/cheza-kernel
  GPU_VERSION: sdm845
  RUNNER_TAG: google-freedreno-cheza
+DEVICE_TYPE: sdm845-cheza-r3
+FARM: google


I see that this is the only user of the FARM: tag. Is it correct?


No, we need to add FARM variable for other jobs as well.


Also we miss DEVICE_TYPE for several other boards. Should we be adding
them?


Yes, device type needs to be added for msm:apq8016, msm:apq8096, virtio_gpu.

I will add this. Thanks.

Regards,
Vignesh




script:
  - ./install/bare-metal/cros-servo.sh
  
--

2.40.1

Re: [PATCH v2 1/6] drm/ci: uprev mesa version

2024-05-20 Thread Dmitry Baryshkov 
On Fri, May 17, 2024 at 02:54:57PM +0530, Vignesh Raman wrote:
> zlib.net is not allowing tarball download anymore and results
> in below error in kernel+rootfs_arm32 container build,
> urllib.error.HTTPError: HTTP Error 403: Forbidden
> urllib.error.HTTPError: HTTP Error 415: Unsupported Media Type
> 
> Uprev mesa to latest version which includes a fix for this issue.
> https://gitlab.freedesktop.org/mesa/mesa/-/commit/908f444e
> 
> Use id_tokens for JWT authentication. Since s3 bucket is migrated to
> mesa-rootfs, update the variables accordingly. Also copy helper scripts
> to install, so that the ci jobs can use these scripts for logging.
> 
> Signed-off-by: Vignesh Raman 
> ---
> 
> v2:
>   - Uprev to recent version and use id_tokens for JWT authentication
> 
> ---
>  drivers/gpu/drm/ci/build-igt.sh   |  2 +-
>  drivers/gpu/drm/ci/build.sh   |  6 +++--
>  drivers/gpu/drm/ci/container.yml  | 12 +++--
>  drivers/gpu/drm/ci/gitlab-ci.yml  | 44 +--
>  drivers/gpu/drm/ci/image-tags.yml |  2 +-
>  drivers/gpu/drm/ci/lava-submit.sh |  4 +--
>  drivers/gpu/drm/ci/test.yml   |  2 ++
>  7 files changed, 44 insertions(+), 28 deletions(-)
> 

[skipped]

> diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
> index 8bc63912fddb..612c9ede3507 100644
> --- a/drivers/gpu/drm/ci/test.yml
> +++ b/drivers/gpu/drm/ci/test.yml
> @@ -150,6 +150,8 @@ msm:sdm845:
>  BM_KERNEL: https://${PIPELINE_ARTIFACTS_BASE}/arm64/cheza-kernel
>  GPU_VERSION: sdm845
>  RUNNER_TAG: google-freedreno-cheza
> +DEVICE_TYPE: sdm845-cheza-r3
> +FARM: google

I see that this is the only user of the FARM: tag. Is it correct?
Also we miss DEVICE_TYPE for several other boards. Should we be adding
them?

>script:
>  - ./install/bare-metal/cros-servo.sh
>  
> -- 
> 2.40.1
> 

-- 
With best wishes
Dmitry

[PATCH v2 1/6] drm/ci: uprev mesa version

2024-05-17 Thread Vignesh Raman 
zlib.net is not allowing tarball download anymore and results
in below error in kernel+rootfs_arm32 container build,
urllib.error.HTTPError: HTTP Error 403: Forbidden
urllib.error.HTTPError: HTTP Error 415: Unsupported Media Type

Uprev mesa to latest version which includes a fix for this issue.
https://gitlab.freedesktop.org/mesa/mesa/-/commit/908f444e

Use id_tokens for JWT authentication. Since s3 bucket is migrated to
mesa-rootfs, update the variables accordingly. Also copy helper scripts
to install, so that the ci jobs can use these scripts for logging.

Signed-off-by: Vignesh Raman 
---

v2:
  - Uprev to recent version and use id_tokens for JWT authentication

---
 drivers/gpu/drm/ci/build-igt.sh   |  2 +-
 drivers/gpu/drm/ci/build.sh   |  6 +++--
 drivers/gpu/drm/ci/container.yml  | 12 +++--
 drivers/gpu/drm/ci/gitlab-ci.yml  | 44 +--
 drivers/gpu/drm/ci/image-tags.yml |  2 +-
 drivers/gpu/drm/ci/lava-submit.sh |  4 +--
 drivers/gpu/drm/ci/test.yml   |  2 ++
 7 files changed, 44 insertions(+), 28 deletions(-)

diff --git a/drivers/gpu/drm/ci/build-igt.sh b/drivers/gpu/drm/ci/build-igt.sh
index 500fa4f5c30a..7859554756c4 100644
--- a/drivers/gpu/drm/ci/build-igt.sh
+++ b/drivers/gpu/drm/ci/build-igt.sh
@@ -32,4 +32,4 @@ tar -cf artifacts/igt.tar /igt
 # Pass needed files to the test stage
 S3_ARTIFACT_NAME="igt.tar.gz"
 gzip -c artifacts/igt.tar > ${S3_ARTIFACT_NAME}
-ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" ${S3_ARTIFACT_NAME} 
https://${PIPELINE_ARTIFACTS_BASE}/${KERNEL_ARCH}/${S3_ARTIFACT_NAME}
+ci-fairy s3cp --token-file "${S3_JWT_FILE}" ${S3_ARTIFACT_NAME} 
https://${PIPELINE_ARTIFACTS_BASE}/${KERNEL_ARCH}/${S3_ARTIFACT_NAME}
diff --git a/drivers/gpu/drm/ci/build.sh b/drivers/gpu/drm/ci/build.sh
index 106f2d40d222..a67871fdcd3f 100644
--- a/drivers/gpu/drm/ci/build.sh
+++ b/drivers/gpu/drm/ci/build.sh
@@ -128,6 +128,7 @@ fi
 # Pass needed files to the test stage
 mkdir -p install
 cp -rfv .gitlab-ci/* install/.
+cp -rfv ci/*  install/.
 cp -rfv install/common install/ci-common
 cp -rfv drivers/gpu/drm/ci/* install/.
 
@@ -141,14 +142,15 @@ if [[ "$UPLOAD_TO_MINIO" = "1" ]]; then
 FILES_TO_UPLOAD="$FILES_TO_UPLOAD $(basename -a $DEVICE_TREES)"
 fi
 
+ls -l "${S3_JWT_FILE}"
 for f in $FILES_TO_UPLOAD; do
-ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" /lava-files/$f \
+ci-fairy s3cp --token-file "${S3_JWT_FILE}" /lava-files/$f \
 https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/$f
 done
 
 S3_ARTIFACT_NAME="kernel-files.tar.zst"
 tar --zstd -cf $S3_ARTIFACT_NAME install
-ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" ${S3_ARTIFACT_NAME} 
https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/${S3_ARTIFACT_NAME}
+ci-fairy s3cp --token-file "${S3_JWT_FILE}" ${S3_ARTIFACT_NAME} 
https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/${S3_ARTIFACT_NAME}
 
 echo "Download vmlinux.xz from 
https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/vmlinux.xz";
 fi
diff --git a/drivers/gpu/drm/ci/container.yml b/drivers/gpu/drm/ci/container.yml
index 9764e7921a4f..d6edf3635b23 100644
--- a/drivers/gpu/drm/ci/container.yml
+++ b/drivers/gpu/drm/ci/container.yml
@@ -36,15 +36,15 @@ debian/android_build:
   rules:
 - when: never
 
-debian/x86_64_test-android:
+.debian/x86_64_test-android:
   rules:
 - when: never
 
-windows_build_vs2019:
+windows_build_msvc:
   rules:
 - when: never
 
-windows_test_vs2019:
+windows_test_msvc:
   rules:
 - when: never
 
@@ -56,10 +56,6 @@ rustfmt:
rules:
 - when: never
 
-windows_vs2019:
-   rules:
-- when: never
-
-clang-format:
+windows_msvc:
rules:
 - when: never
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml
index 084e3ff8e3f4..8f32de63d92e 100644
--- a/drivers/gpu/drm/ci/gitlab-ci.yml
+++ b/drivers/gpu/drm/ci/gitlab-ci.yml
@@ -1,6 +1,6 @@
 variables:
   DRM_CI_PROJECT_PATH: &drm-ci-project-path mesa/mesa
-  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha 
9d162de9a05155e1c4041857a5848842749164cf
+  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha 
e2b9c5a9e3e4f9b532067af8022eaef8d6fc6c00
 
   UPSTREAM_REPO: git://anongit.freedesktop.org/drm/drm
   TARGET_BRANCH: drm-next
@@ -19,33 +19,47 @@ variables:
   bash download-git-cache.sh
   rm download-git-cache.sh
   set +o xtrace
+  S3_JWT_FILE: /s3_jwt
   S3_HOST: s3.freedesktop.org
+  # This bucket is used to fetch the kernel image
+  S3_KERNEL_BUCKET: mesa-rootfs
+  # Bucket for git cache
+  S3_GITCACHE_BUCKET: git-cache
+  # Bucket for the pipeline artifacts pushed to S3
+  S3_ARTIFACTS_BUCKET: artifacts
   # per-pipeline artifact storage on MinIO
-  PIPELINE_ARTIFACTS_BASE: 
${S3_HOST}/artifacts/${CI_PROJECT_PATH}/${CI_PIPELINE_ID}
+  PIPELINE_ARTIFACTS_BASE: 
${S3_HOST}/${S3_ARTIFACTS_BUCKET}/${CI_PROJECT_PATH}/${CI_PIPELINE_ID}
   # per-job artifact storage on MinIO
   JOB_ARTIFACTS_BASE: ${PIPELINE_ARTIFACTS_BASE