[ https://issues.apache.org/jira/browse/HDDS-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siyao Meng updated HDDS-6263: ----------------------------- Description: This is the uber jira for arm64 support in Ozone docker development environment. Status Quo: - Most images used in docker dev environment supports amd64 only. - It runs VERY slowly on Apple Silicon (M1) Macs' Docker Desktop due to qemu x86-64 -> arm64 translation. For instance, SCM startup in the upgrade acceptance test should take ~23 seconds on a regular Intel MacBook Pro to boot up and exit safe mode; while it times out (after 240s~300s) starting up the SCM alone on an M1 Pro MacBook Pro due to the translation/emulation. Goal: - Add arm64 support for all Docker images used for Ozone development. The resulting images (that are pushed to Docker Hub) should be dual-architecture (e.g. ozone-docker-runner). Foreseeable work to be done: - Bump Centos base image to a version that (Almost done, pending review) - dumbinit and tools included in the image build stage should be architecture aware. Currently the tools are x64 only. After this the ozone runner base image should be working, provided that ALL the Ozone dependencies at runtime that uses native library pulled from maven supports Linux arm64/aarch64 (afaik RocksDBJNI does, gRPC also [does|https://repo1.maven.org/maven2/io/grpc/protoc-gen-grpc-java/1.33.0/], newer protobuf 3 should also have it, but older protobuf 2.5.0 might not. However, it seems gRPC and protobuf 2/3 doesn't matter at the moment as they are only used at compile time but not runtime.) - Have arm64 version of Hadoop 3 / KMS / KDC images as well, which are referenced in some testing environment like ozonesecure for KDC. - The last point might not be strictly required as IIRC Docker itself supports running some containers natively but others in emulation at the same time. - Hadoop 3 already has the arm64 Dockerfile ready. But no one has built and pushed an arm64 image to Docker Hub yet: - https://hub.docker.com/r/apache/hadoop/tags - https://hub.docker.com/r/apache/hadoop-runner/tags was: This is the uber jira for arm64 support in Ozone docker development environment. Status Quo: - Most images used in docker dev environment supports amd64 only. - It runs VERY slowly on Apple Silicon (M1) Macs' Docker Desktop due to qemu x86-64 -> arm64 translation. For instance, SCM startup in the upgrade acceptance test should take ~23 seconds on a regular Intel MacBook Pro to boot up and exit safe mode; while it times out (after 240s~300s) starting up the SCM alone on an M1 Pro MacBook Pro due to the translation/emulation. Goal: - Add arm64 support for all Docker images used for Ozone development. The resulting images (that are pushed to Docker Hub) should be dual-architecture (e.g. ozone-docker-runner). Foreseeable work to be done: - Bump Centos base image to a version that (Almost done, pending review) - dumbinit and tools included in the image build stage should be architecture aware. Currently the tools are x64 only. After this the ozone runner base image should be working, provided that ALL the Ozone dependencies pulled from maven supports Linux arm64/aarch64 (afaik RocksDBJNI does, gRPC also [does|https://repo1.maven.org/maven2/io/grpc/protoc-gen-grpc-java/1.33.0/], newer protobuf 3 should also have it, but older protobuf 2.5.0 might not) - Have arm64 version of Hadoop 3 / KMS / KDC images as well, which are referenced in some testing environment like ozonesecure for KDC. - The last point might not be strictly required as IIRC Docker itself supports running some containers natively but others in emulation at the same time. - Hadoop 3 already has the arm64 Dockerfile ready. But no one has built and pushed an arm64 image to Docker Hub yet: - https://hub.docker.com/r/apache/hadoop/tags - https://hub.docker.com/r/apache/hadoop-runner/tags > Full arm64 support for Docker Images > ------------------------------------ > > Key: HDDS-6263 > URL: https://issues.apache.org/jira/browse/HDDS-6263 > Project: Apache Ozone > Issue Type: Task > Components: docker > Reporter: Siyao Meng > Priority: Major > > This is the uber jira for arm64 support in Ozone docker development > environment. > Status Quo: > - Most images used in docker dev environment supports amd64 only. > - It runs VERY slowly on Apple Silicon (M1) Macs' Docker Desktop due to qemu > x86-64 -> arm64 translation. For instance, SCM startup in the upgrade > acceptance test should take ~23 seconds on a regular Intel MacBook Pro to > boot up and exit safe mode; while it times out (after 240s~300s) starting up > the SCM alone on an M1 Pro MacBook Pro due to the translation/emulation. > Goal: > - Add arm64 support for all Docker images used for Ozone development. The > resulting images (that are pushed to Docker Hub) should be dual-architecture > (e.g. ozone-docker-runner). > Foreseeable work to be done: > - Bump Centos base image to a version that (Almost done, pending review) > - dumbinit and tools included in the image build stage should be architecture > aware. Currently the tools are x64 only. After this the ozone runner base > image should be working, provided that ALL the Ozone dependencies at runtime > that uses native library pulled from maven supports Linux arm64/aarch64 > (afaik RocksDBJNI does, gRPC also > [does|https://repo1.maven.org/maven2/io/grpc/protoc-gen-grpc-java/1.33.0/], > newer protobuf 3 should also have it, but older protobuf 2.5.0 might not. > However, it seems gRPC and protobuf 2/3 doesn't matter at the moment as they > are only used at compile time but not runtime.) > - Have arm64 version of Hadoop 3 / KMS / KDC images as well, which are > referenced in some testing environment like ozonesecure for KDC. > - The last point might not be strictly required as IIRC Docker itself > supports running some containers natively but others in emulation at the same > time. > - Hadoop 3 already has the arm64 Dockerfile ready. But no one has built and > pushed an arm64 image to Docker Hub yet: > - https://hub.docker.com/r/apache/hadoop/tags > - https://hub.docker.com/r/apache/hadoop-runner/tags -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org For additional commands, e-mail: issues-h...@ozone.apache.org