For awareness with the just released Beam 2.36.0 Beam works out of the box to develop on a Mac M1.
I tried Java and Python pipelines with success running locally on both Flink/Spark runner. I found one issue using zstd and created [1] that was merged today, with this the sdks:core tests and Spark runner tests fully pass. I would see 2.36.0 is the first good enough release for someone working on a Mac M1 or ARM64 processor. There are still some missing steps to have full ARM64 [apart of testing it :)] 1. In theory we could run docker x86 images on ARM but those would be emulated so way slower so it is probably better to support 'native' CPUs) via multiarchitecture docker images [2]. BEAM-11704 Support Beam docker images on ARM64 I could create the runners images from master, for the SDK containers there are some issues with hardcoded paths [2] and virtualenv that probably will be solved once we move to venv, and we will need to upgrade our release process to include multiarch images (for user friendliness). Also golang only supports officially ARM64 starting with version 1.18.0 so we need to move up to that version. Anyway Beam is in a waaay better shape for ARM64 now than 1y ago when I created the initial JIRAs. Ismaël [1] https://github.com/apache/beam/pull/16755 [2] https://issues.apache.org/jira/browse/BEAM-11704 [3] https://github.com/apache/beam/blob/d1b8e569fd651975f08823a3db49dbee56d491b5/sdks/python/container/Dockerfile#L79 > Could not find protoc-3.14.0-osx-aarch_64.exe (com.google.protobuf:protoc:3.14.0). Searched in the following locations: https://jcenter.bintray.com/com/google/protobuf/protoc/3.14.0/protoc-3.14.0-osx-aarch_64.exe On Wed, Jan 12, 2022 at 9:53 PM Luke Cwik <lc...@google.com> wrote: > > The docker container running in an x86 based cloud machine should work pretty > well. This is what Apache Beam's Jenkins setup effectively does. > > No experience with developing on an ARM based CPU. > > On Wed, Jan 12, 2022 at 9:28 AM Jarek Potiuk <ja...@potiuk.com> wrote: >> >> Comment from the side - If you use Docker - experience from Airflow - >> until we will get ARM images, docker experience is next to unusable >> (docker filesystem slowness + emulation). >> >> J. >> >> On Wed, Jan 12, 2022 at 6:21 PM Daniel Collins <dpcoll...@google.com> wrote: >> > >> > I regularly develop on a non-m1 mac using intellij, which mostly works out >> > of the box. Are you running into any particular issues building or just >> > looking for advice? >> > >> > -Daniel >> > >> > On Wed, Jan 12, 2022 at 12:16 PM Matt Rudary <matt.rud...@twosigma.com> >> > wrote: >> >> >> >> Does anyone do Beam development on an M1 Mac? Any tips to getting things >> >> up and running? >> >> >> >> >> >> >> >> Alternatively, does anyone have a good “workstation in the cloud” setup? >> >> >> >> >> >> >> >> Thanks >> >> >> >> Matt
