Thanks for pointing this out Robert, I had somehow in my mind that it was not official until 1.18 but I forgot to double check, for info I was able to build the Go SDK container without any trouble so I assume most of the things are 'ready'.
It would be great to take a look at fixing the hardcoded arch uses in paths and other (grep -i -R 'amd64') , don't hesitate to ping me if you need help with some some test I have easy acces to AWS Graviton too. Of course the Beam community has never mentioned being ARM64 compatible as a target but with the increasing rise of this arch on the server side and now on the Mac, it would be probably worth. In case we decide to do it, Github Actions does not support ARM64 yet, so other projects validate ARM64 through emulation via qemu, but this seems to be quite slow. On Tue, Feb 8, 2022 at 3:58 PM Robert Burke <[email protected]> wrote: > > Go supports ARM64 on Darwin since 1.16, which is the minimum version of Go we > currently support. > > See https://go.dev/blog/ports > > There are definitely some hardcoded paths we'd need to adjust to build boot > containers though. > > Go 1.18 improves things, and since it has the initial run of Go Generics, > we'll likely move to support it pretty quickly. > > On Tue, Feb 8, 2022, 6:18 AM Jarek Potiuk <[email protected]> wrote: >> >> Just for your information: Thanks to that change - i will soon be adding ARM >> support for Apache Airflow - including building and publishing the images >> and running our tests (using self-hosted runners). >> As soon as I get it I will be able to share the code/experiences with you. >> >> J >> >> On Tue, Feb 8, 2022 at 2:50 PM Ismaël Mejía <[email protected]> wrote: >>> >>> For awareness with the just released Beam 2.36.0 Beam works out of the >>> box to develop on a Mac M1. >>> >>> I tried Java and Python pipelines with success running locally on both >>> Flink/Spark runner. >>> I found one issue using zstd and created [1] that was merged today, >>> with this the sdks:core tests and Spark runner tests fully pass. >>> >>> I would see 2.36.0 is the first good enough release for someone >>> working on a Mac M1 or ARM64 processor. >>> >>> There are still some missing steps to have full ARM64 [apart of testing it >>> :)] >>> >>> 1. In theory we could run docker x86 images on ARM but those would be >>> emulated so way slower so it is probably better to support 'native' >>> CPUs) via multiarchitecture docker images [2]. >>> BEAM-11704 Support Beam docker images on ARM64 >>> >>> I could create the runners images from master, for the SDK containers >>> there are some issues with hardcoded paths [2] and virtualenv that >>> probably will be solved once we move to venv, and we will need to >>> upgrade our release process to include multiarch images (for user >>> friendliness). >>> >>> Also golang only supports officially ARM64 starting with version >>> 1.18.0 so we need to move up to that version. >>> >>> Anyway Beam is in a waaay better shape for ARM64 now than 1y ago when >>> I created the initial JIRAs. >>> >>> Ismaël >>> >>> [1] https://github.com/apache/beam/pull/16755 >>> [2] https://issues.apache.org/jira/browse/BEAM-11704 >>> [3] >>> https://github.com/apache/beam/blob/d1b8e569fd651975f08823a3db49dbee56d491b5/sdks/python/container/Dockerfile#L79 >>> >>> >>> >>> > Could not find protoc-3.14.0-osx-aarch_64.exe >>> (com.google.protobuf:protoc:3.14.0). >>> Searched in the following locations: >>> >>> https://jcenter.bintray.com/com/google/protobuf/protoc/3.14.0/protoc-3.14.0-osx-aarch_64.exe >>> >>> >>> >>> >>> >>> On Wed, Jan 12, 2022 at 9:53 PM Luke Cwik <[email protected]> wrote: >>> > >>> > The docker container running in an x86 based cloud machine should work >>> > pretty well. This is what Apache Beam's Jenkins setup effectively does. >>> > >>> > No experience with developing on an ARM based CPU. >>> > >>> > On Wed, Jan 12, 2022 at 9:28 AM Jarek Potiuk <[email protected]> wrote: >>> >> >>> >> Comment from the side - If you use Docker - experience from Airflow - >>> >> until we will get ARM images, docker experience is next to unusable >>> >> (docker filesystem slowness + emulation). >>> >> >>> >> J. >>> >> >>> >> On Wed, Jan 12, 2022 at 6:21 PM Daniel Collins <[email protected]> >>> >> wrote: >>> >> > >>> >> > I regularly develop on a non-m1 mac using intellij, which mostly works >>> >> > out of the box. Are you running into any particular issues building or >>> >> > just looking for advice? >>> >> > >>> >> > -Daniel >>> >> > >>> >> > On Wed, Jan 12, 2022 at 12:16 PM Matt Rudary >>> >> > <[email protected]> wrote: >>> >> >> >>> >> >> Does anyone do Beam development on an M1 Mac? Any tips to getting >>> >> >> things up and running? >>> >> >> >>> >> >> >>> >> >> >>> >> >> Alternatively, does anyone have a good “workstation in the cloud” >>> >> >> setup? >>> >> >> >>> >> >> >>> >> >> >>> >> >> Thanks >>> >> >> >>> >> >> Matt
