I believe so. The Go SDK requires in most instances for a user to Register their DoFns at package init time, linked to the type/functions fully qualified path as detemined by Go, which is consistent across architectures, at least with the standard toochain.
Those strings are used to look things up on distributed workers, regardless of the architecture. On Tue, Jan 26, 2021, 11:33 AM Robert Bradshaw <rober...@google.com> wrote: > Cool. Are DoFn (et al) references compatible across cross-compiled > binaries? > > On Tue, Jan 26, 2021 at 11:23 AM Robert Burke <rob...@frantil.com> wrote: > >> Go cross compilation is as simple as setting the right flag env variables >> [1], but can be as complicated as requiring a cross compiling GCC instance >> installed if CGO[2] is necessary. I think we're probably clear on just >> needing the flag though for the various Boot executables. >> >> For go pipelines we'd need to update the shared runner code to support >> selecting the cross compiled worker binary environment. I believe it's hard >> set to amd64 linux at present, but that's a separate issue. >> >> [1] https://golangcookbook.com/chapters/running/cross-compiling/ >> [2] https://golang.org/cmd/cgo/ >> >> On Tue, Jan 26, 2021, 10:25 AM Robert Bradshaw <rober...@google.com> >> wrote: >> >>> +1 >>> >>> I don't think it would be that hard to build and release arm-based >>> docker images. (Perhaps just a matter of changing the docker file to depend >>> on a different base, and doing some cross-compile. That would suss out >>> whether we're inadvertently taking on any incompatible dependencies.) >>> >>> Theoretically, if one does that and manually specifies the container, it >>> could just work for Python (assuming no wheel files are specified as manual >>> dependencies). For Java, if one builds/deploys an uberjar (on a different >>> architecture), there may be issues in any transitive dependency that has >>> JNI code (us or users). I'd imagine this issue is common to and being >>> explored by many of the other Java big data systems in use; it'd be >>> interesting to know what solutions are out there. >>> >>> For go, the executable is uploaded directly into the container. We'd >>> probably have to do something fancier like cross-compiling the executable >>> (and making sure the UserFn references, which I think are just pointers >>> into the binary, still work if the launcher is one architecture and the >>> workers another). >>> >>> Definitely worth exploring. >>> >>> >>> >>> >>> On Tue, Jan 26, 2021 at 10:09 AM Ismaël Mejía <ieme...@gmail.com> wrote: >>> >>>> I stumbled today on this user request: >>>> BEAM-10982 Wheel support for linux aarch64 >>>> >>>> It made me wonder if with the advent of ARM64 processors not only in >>>> the client but server side (Graviton and others) if it is worth that >>>> we start to think about having support for this architecture on the >>>> python installers and in the docker images. It seems that for the >>>> latter it should not be that difficult given that our parent images >>>> are already multi-arch. >>>> >>>> Are there some possible issues or binary/platform specific >>>> dependencies that impede us from doing this? >>>> >>>
