Nice to see the interest, I also suppose that devs on Apple macbooks with the
new M1 processor will soon request this feature.
I ran today some pipelines on ARM64 on classic runners relatively easy
which was expected. We will have issues however for the Java 8 SDK harness
because the parent image openjdk:8 is not supported yet for ARM64.
I tried to setup a python dev environment and found the first issue. It looks
like gRPC does not support arm64 yet [1][2] or am I misreading it?
$ pip install -r build-requirements.txt
Collecting grpcio-tools==1.30.0
Downloading grpcio-tools-1.30.0.tar.gz (2.1 MB)
|████████████████████████████████| 2.1 MB 21.7 MB/s
ERROR: Command errored out with exit status 1:
command: /home/ubuntu/.virtualenvs/beam-dev/bin/python3 -c
'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-3lhad2qc/grpcio-tools_d3562157df5c41db9110e4ccd165c87e/setup.py'"'"';
__file__='"'"'/tmp/pip-install-3lhad2qc/grpcio-tools_d3562157df5c41db9110e4ccd165c87e/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
egg_info --egg-base /tmp/pip-pip-egg-info-km8agjf4
cwd:
/tmp/pip-install-3lhad2qc/grpcio-tools_d3562157df5c41db9110e4ccd165c87e/
Complete output (11 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File
"/tmp/pip-install-3lhad2qc/grpcio-tools_d3562157df5c41db9110e4ccd165c87e/setup.py",
line 112, in <module>
if check_linker_need_libatomic():
File
"/tmp/pip-install-3lhad2qc/grpcio-tools_d3562157df5c41db9110e4ccd165c87e/setup.py",
line 73, in check_linker_need_libatomic
cc_test = subprocess.Popen(['cc', '-x', 'c++', '-std=c++11', '-'],
File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'cc'
----------------------------------------
WARNING: Discarding
https://files.pythonhosted.org/packages/da/3c/bed275484f6cc262b5de6ceaae36798c60d7904cdd05dc79cc830b880687/grpcio-tools-1.30.0.tar.gz#sha256=7878adb93b0c1941eb2e0bed60719f38cda2ae5568bc0bcaa701f457e719a329
(from https://pypi.org/simple/grpcio-tools/). Command errored out with
exit status 1: python setup.py egg_info Check the logs for full
command output.
ERROR: Could not find a version that satisfies the requirement
grpcio-tools==1.30.0
ERROR: No matching distribution found for grpcio-tools==1.30.0
[1] https://pypi.org/project/grpcio-tools/#files
[2] https://github.com/grpc/grpc/issues/21283
I can imagine also that we will have some struggles with the python harness
and all of its dependencies. Does cython already support ARM64?
I went and filled some JIRAs to keep track of this:
BEAM-11703 Support apache-beam python install on ARM64
BEAM-11704 Support Beam docker images on ARM64
On Tue, Jan 26, 2021 at 8:48 PM Robert Burke <rob...@frantil.com> wrote:
>
> I believe so.
>
> The Go SDK requires in most instances for a user to Register their DoFns at
> package init time, linked to the type/functions fully qualified path as
> detemined by Go, which is consistent across architectures, at least with the
> standard toochain.
>
> Those strings are used to look things up on distributed workers, regardless
> of the architecture.
>
>
>
> On Tue, Jan 26, 2021, 11:33 AM Robert Bradshaw <rober...@google.com> wrote:
>>
>> Cool. Are DoFn (et al) references compatible across cross-compiled binaries?
>>
>> On Tue, Jan 26, 2021 at 11:23 AM Robert Burke <rob...@frantil.com> wrote:
>>>
>>> Go cross compilation is as simple as setting the right flag env variables
>>> [1], but can be as complicated as requiring a cross compiling GCC instance
>>> installed if CGO[2] is necessary. I think we're probably clear on just
>>> needing the flag though for the various Boot executables.
>>>
>>> For go pipelines we'd need to update the shared runner code to support
>>> selecting the cross compiled worker binary environment. I believe it's hard
>>> set to amd64 linux at present, but that's a separate issue.
>>>
>>> [1] https://golangcookbook.com/chapters/running/cross-compiling/
>>> [2] https://golang.org/cmd/cgo/
>>>
>>> On Tue, Jan 26, 2021, 10:25 AM Robert Bradshaw <rober...@google.com> wrote:
>>>>
>>>> +1
>>>>
>>>> I don't think it would be that hard to build and release arm-based docker
>>>> images. (Perhaps just a matter of changing the docker file to depend on a
>>>> different base, and doing some cross-compile. That would suss out whether
>>>> we're inadvertently taking on any incompatible dependencies.)
>>>>
>>>> Theoretically, if one does that and manually specifies the container, it
>>>> could just work for Python (assuming no wheel files are specified as
>>>> manual dependencies). For Java, if one builds/deploys an uberjar (on a
>>>> different architecture), there may be issues in any transitive dependency
>>>> that has JNI code (us or users). I'd imagine this issue is common to and
>>>> being explored by many of the other Java big data systems in use; it'd be
>>>> interesting to know what solutions are out there.
>>>>
>>>> For go, the executable is uploaded directly into the container. We'd
>>>> probably have to do something fancier like cross-compiling the executable
>>>> (and making sure the UserFn references, which I think are just pointers
>>>> into the binary, still work if the launcher is one architecture and the
>>>> workers another).
>>>>
>>>> Definitely worth exploring.
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Jan 26, 2021 at 10:09 AM Ismaël Mejía <ieme...@gmail.com> wrote:
>>>>>
>>>>> I stumbled today on this user request:
>>>>> BEAM-10982 Wheel support for linux aarch64
>>>>>
>>>>> It made me wonder if with the advent of ARM64 processors not only in
>>>>> the client but server side (Graviton and others) if it is worth that
>>>>> we start to think about having support for this architecture on the
>>>>> python installers and in the docker images. It seems that for the
>>>>> latter it should not be that difficult given that our parent images
>>>>> are already multi-arch.
>>>>>
>>>>> Are there some possible issues or binary/platform specific
>>>>> dependencies that impede us from doing this?
