Re: Multiple architectures support on Beam (ARM)

2021-06-10 Thread Robert Bradshaw
On Thu, Jun 10, 2021 at 3:00 AM Ismaël Mejía wrote: > > As a follow up on this with the merge of > https://github.com/apache/beam/pull/14832 Beam will be producing python > wheels for AARCH64 starting on Beam 2.32.0! Nice. > Also due to the recent version updates (grpc, protobuf and arrow) we

Re: Multiple architectures support on Beam (ARM)

2021-06-10 Thread Ismaël Mejía
As a follow up on this with the merge of https://github.com/apache/beam/pull/14832 Beam will be producing python wheels for AARCH64 starting on Beam 2.32.0! Also due to the recent version updates (grpc, protobuf and arrow) we should be pretty close to fully support it without extra compilation.

Re: Multiple architectures support on Beam (ARM)

2021-01-27 Thread Robert Bradshaw
Cython supports ARM64. The issue here is that we don't have a C++ compiler (It's looking for 'cc') available in the container (and grpc, and possibly others, don't have wheel files for this platform). I wonder if apt-get install build-essential would be sufficient. On Wed, Jan 27, 2021 at 2:22 PM

Re: Multiple architectures support on Beam (ARM)

2021-01-27 Thread Ismaël Mejía
Nice to see the interest, I also suppose that devs on Apple macbooks with the new M1 processor will soon request this feature. I ran today some pipelines on ARM64 on classic runners relatively easy which was expected. We will have issues however for the Java 8 SDK harness because the parent

Re: Multiple architectures support on Beam (ARM)

2021-01-26 Thread Robert Burke
I believe so. The Go SDK requires in most instances for a user to Register their DoFns at package init time, linked to the type/functions fully qualified path as detemined by Go, which is consistent across architectures, at least with the standard toochain. Those strings are used to look things

Re: Multiple architectures support on Beam (ARM)

2021-01-26 Thread Robert Bradshaw
Cool. Are DoFn (et al) references compatible across cross-compiled binaries? On Tue, Jan 26, 2021 at 11:23 AM Robert Burke wrote: > Go cross compilation is as simple as setting the right flag env variables > [1], but can be as complicated as requiring a cross compiling GCC instance > installed

Re: Multiple architectures support on Beam (ARM)

2021-01-26 Thread Robert Burke
Go cross compilation is as simple as setting the right flag env variables [1], but can be as complicated as requiring a cross compiling GCC instance installed if CGO[2] is necessary. I think we're probably clear on just needing the flag though for the various Boot executables. For go pipelines

Re: Multiple architectures support on Beam (ARM)

2021-01-26 Thread Robert Bradshaw
+1 I don't think it would be that hard to build and release arm-based docker images. (Perhaps just a matter of changing the docker file to depend on a different base, and doing some cross-compile. That would suss out whether we're inadvertently taking on any incompatible dependencies.)

Multiple architectures support on Beam (ARM)

2021-01-26 Thread Ismaël Mejía
I stumbled today on this user request: BEAM-10982 Wheel support for linux aarch64 It made me wonder if with the advent of ARM64 processors not only in the client but server side (Graviton and others) if it is worth that we start to think about having support for this architecture on the python