[jira] [Commented] (YUNIKORN-725) Support arm64
[ https://issues.apache.org/jira/browse/YUNIKORN-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17538369#comment-17538369 ] Craig Condit commented on YUNIKORN-725: --- With the first 4 subtasks defined here, we should be in a state where we can successfully build both local and docker images for both amd64 and arm64 (though not cross-architecture). I think we should open up a separate JIRA entirely for how we might be able to produce multi-arch binaries for release builds. I'm not overly concerned about our ability to do that currently, as the binaries we produce are for convenience and users should probably build their own from source regardless. Thoughts? > Support arm64 > - > > Key: YUNIKORN-725 > URL: https://issues.apache.org/jira/browse/YUNIKORN-725 > Project: Apache YuniKorn > Issue Type: Improvement > Components: release >Reporter: Holden Karau >Priority: Major > > It would be good to support arm64, this is probably not too painful and can > be done by swapping docker build with docker buildx, but there's often edge > cases where some code changes are needed. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
[jira] [Commented] (YUNIKORN-725) Support arm64
[ https://issues.apache.org/jira/browse/YUNIKORN-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498359#comment-17498359 ] Weiwei Yang commented on YUNIKORN-725: -- That's fine, thanks [~srisco]. Thanks for the remark as well : ) > Support arm64 > - > > Key: YUNIKORN-725 > URL: https://issues.apache.org/jira/browse/YUNIKORN-725 > Project: Apache YuniKorn > Issue Type: Improvement > Components: release >Reporter: Holden Karau >Priority: Major > > It would be good to support arm64, this is probably not too painful and can > be done by swapping docker build with docker buildx, but there's often edge > cases where some code changes are needed. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
[jira] [Commented] (YUNIKORN-725) Support arm64
[ https://issues.apache.org/jira/browse/YUNIKORN-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498023#comment-17498023 ] Sebastián R commented on YUNIKORN-725: -- Hi [~wwei] and [~wilfreds], sorry for the late reply. Sorry, I have a very busy next weeks writing my PhD thesis and also I'm not very used to Makefiles Therefore I would step aside for this issue. Thanks for counting on me anyway, I hope to contribute in the future. One last point I would like to remark is that probably you also need buildx to generate the images even choosing the manifest approach. This is because the base images also have to be matched to a particular platform and I'm not sure if docker build supports downloading the appropriate multi-arch image based on --build-arg ARCH=xxx. > Support arm64 > - > > Key: YUNIKORN-725 > URL: https://issues.apache.org/jira/browse/YUNIKORN-725 > Project: Apache YuniKorn > Issue Type: Improvement > Components: release >Reporter: Holden Karau >Priority: Major > > It would be good to support arm64, this is probably not too painful and can > be done by swapping docker build with docker buildx, but there's often edge > cases where some code changes are needed. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
[jira] [Commented] (YUNIKORN-725) Support arm64
[ https://issues.apache.org/jira/browse/YUNIKORN-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497183#comment-17497183 ] Weiwei Yang commented on YUNIKORN-725: -- hi [~wilfreds] all good points, thanks. I agree we can go with the option using docker manifest. [~srisco] not sure if you want to take this and move on to the next step if you agree with this approach? > Support arm64 > - > > Key: YUNIKORN-725 > URL: https://issues.apache.org/jira/browse/YUNIKORN-725 > Project: Apache YuniKorn > Issue Type: Improvement > Components: release >Reporter: Holden Karau >Priority: Major > > It would be good to support arm64, this is probably not too painful and can > be done by swapping docker build with docker buildx, but there's often edge > cases where some code changes are needed. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
[jira] [Commented] (YUNIKORN-725) Support arm64
[ https://issues.apache.org/jira/browse/YUNIKORN-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17495901#comment-17495901 ] Wilfred Spiegelenburg commented on YUNIKORN-725: We never release from github actions. Binary convenience releases are build from the release tar ball. They *must* be build on hardware the [release manager has control over|http://www.apache.org/legal/release-policy.html#owned-controlled-hardware]. That means that unless we have something that any release manager can run, locally, as part of the release process we should not support it. It does not mean that quality testing needs to happen on the local machine. I do not think building inside a docker image is the way to go. It adds a lot of overhead. We also do not use the native go cross compilation. The docker buildx option will use QEMU, an emulation. Support for cross compilation from go is first class so why rely on an emulation layer instead of just directly the compiler. Docker desktop provides a simple way to do all this. However we cannot expect the developer to have docker desktop as that might clash with what the developer can or will install. On a linux machine it could become tricky unless you use a distro that has done all the hard work for you. Using go and plain docker works out of the box. Building a manifest is a once of task. We figure out which command to run and we can generate it over and over from the release tool. This means we reuse the command with the correct flags every time we need to build a manifest. The reason I prefer that option is that we will have changes in the near future. Apple MacBook Pro has already moved to an ARM processor. Lots of the developers in the community are on that platform. We need to be able to support a mixed developer community better. We will eventually have binary releases for AMD/Intel that have been cross compiled on a ARM based machine. > Support arm64 > - > > Key: YUNIKORN-725 > URL: https://issues.apache.org/jira/browse/YUNIKORN-725 > Project: Apache YuniKorn > Issue Type: Improvement > Components: release >Reporter: Holden Karau >Priority: Major > > It would be good to support arm64, this is probably not too painful and can > be done by swapping docker build with docker buildx, but there's often edge > cases where some code changes are needed. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
[jira] [Commented] (YUNIKORN-725) Support arm64
[ https://issues.apache.org/jira/browse/YUNIKORN-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17495871#comment-17495871 ] Weiwei Yang commented on YUNIKORN-725: -- Thanks [~srisco], [~wilfreds]. Sorry for the late response, I was on vacation. I don't have much experience with the cross-compiling stuff, it looks like there are 2 options, docker buildx, and docker manifest. The buildx approach looks so much easier, why do you prefer the manifest approach [~wilfreds]? Except for the build script changes, how can we make sure the image we built is working well on arm64 hosts? As far as I can tell, github action doesn't support arm64 which means we won't be able to have automated tests for that. I am not sure what is the best practice for this, any suggestions? > Support arm64 > - > > Key: YUNIKORN-725 > URL: https://issues.apache.org/jira/browse/YUNIKORN-725 > Project: Apache YuniKorn > Issue Type: Improvement > Components: release >Reporter: Holden Karau >Priority: Major > > It would be good to support arm64, this is probably not too painful and can > be done by swapping docker build with docker buildx, but there's often edge > cases where some code changes are needed. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
[jira] [Commented] (YUNIKORN-725) Support arm64
[ https://issues.apache.org/jira/browse/YUNIKORN-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17494019#comment-17494019 ] Wilfred Spiegelenburg commented on YUNIKORN-725: I would opt for the cross compiling with go and building a manifest. The manifest generation we can easily add to our automation tool for the release creation. We should fix up the Makefile in the k8shim to pick up the GOOS and GOARCH settings from outside the file. That allows us to override the settings and cross compile. We can use the same architecture value to pass into the docker build. Just build one binary as part of the normal tests etc. Multiple binaries and images for release triggered from the release tools. With the copy of the binary we already d we should be able to use one docker file also as we can rename the different architecture builds to the same executable in the image. > Support arm64 > - > > Key: YUNIKORN-725 > URL: https://issues.apache.org/jira/browse/YUNIKORN-725 > Project: Apache YuniKorn > Issue Type: Improvement >Reporter: Holden Karau >Priority: Major > > It would be good to support arm64, this is probably not too painful and can > be done by swapping docker build with docker buildx, but there's often edge > cases where some code changes are needed. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
[jira] [Commented] (YUNIKORN-725) Support arm64
[
https://issues.apache.org/jira/browse/YUNIKORN-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493843#comment-17493843
]
Sebastián R commented on YUNIKORN-725:
--
Hi [~wwei], as [~wilfreds] mentioned, it's not necessary to have a new
environment to build multi-arch images. However, I think it's important to
clarify some points:
* Currently the binaries are compiled before the images (not in the same
Dockerfile), so in the docker build process it only copies the binary inside
the image.
* With docker buildx, using a single dockerfile you get a multi-arch image,
supporting multiple platforms {*}with the same tag{*}.
* If the idea is to keep generating the binaries by cross-compiling with go
you would need to have different dockerfiles for each architecture and in order
to get multi-arch images you would need to unify them by using docker manifest.
This
[post|https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/]
explains the different approaches.
I have
[experience|https://github.com/grycap/oscar/blob/master/.github/workflows/main.yaml#L27]
working with docker buildx compiling the binaries within the image creation
process (it can be done on another stage), but it is true that the building is
slower since it uses qemu to emulate the different architectures. Moreover,
cross-compiling with go is faster, but creating a multi-arch image is more
tedious. So I think the first step is to decide which approach to use.
> Support arm64
> -
>
> Key: YUNIKORN-725
> URL: https://issues.apache.org/jira/browse/YUNIKORN-725
> Project: Apache YuniKorn
> Issue Type: Improvement
>Reporter: Holden Karau
>Priority: Major
>
> It would be good to support arm64, this is probably not too painful and can
> be done by swapping docker build with docker buildx, but there's often edge
> cases where some code changes are needed.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
[jira] [Commented] (YUNIKORN-725) Support arm64
[ https://issues.apache.org/jira/browse/YUNIKORN-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493551#comment-17493551 ] Wilfred Spiegelenburg commented on YUNIKORN-725: We do not need a new environment for this. With the [docker buildx|https://docs.docker.com/buildx/working-with-buildx/] tool as pointed out by Holden we can do this from a linux host or even Mac laptop. We just need to update the build commands for the images. The docker blog describes this [here|https://www.docker.com/blog/multi-platform-docker-builds/]. Docker Desktop for Mac comes with docker buildx and is free to use for a developer working on an open source project (**caveats apply based on your employer ect) For us that means we need to: * generate a binary using a cross compilation by specifying GOOS & GOARCH * update the Makefile to use buildx * build on a machine that has buildx, for both release and github actions The current PR workflow uses the latest ubuntu release as the OS. According to github action tool support for [ubuntu latest|https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-Readme.md] we have docker buildx as a build tool available. It should thus not break our PR workflow either. We only need to generate multiple docker images during the release cycle. Allowing a user to build an image for another platform from the source release should also be possible. So as long a the config in the shim allows us to easily do that and does not break our PR workflow I do not see why we cannot build that into the base Makefiles. We do need some help with this [~srisco]. I assume there are more people around that could help out if we get stuck somewhere. > Support arm64 > - > > Key: YUNIKORN-725 > URL: https://issues.apache.org/jira/browse/YUNIKORN-725 > Project: Apache YuniKorn > Issue Type: Improvement >Reporter: Holden Karau >Priority: Major > > It would be good to support arm64, this is probably not too painful and can > be done by swapping docker build with docker buildx, but there's often edge > cases where some code changes are needed. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
[jira] [Commented] (YUNIKORN-725) Support arm64
[ https://issues.apache.org/jira/browse/YUNIKORN-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493470#comment-17493470 ] Weiwei Yang commented on YUNIKORN-725: -- hi [~srisco] I do not know anyone who has an env to create or test arm64 images, do you want to give it a try? We are building our docker images with the Makefile: https://github.com/apache/incubator-yunikorn-k8shim/blob/1966efe93dcc1eddadf45a35a01830cedaff6a35/Makefile#L270-L271, all docker files are in the repo. Thanks > Support arm64 > - > > Key: YUNIKORN-725 > URL: https://issues.apache.org/jira/browse/YUNIKORN-725 > Project: Apache YuniKorn > Issue Type: Improvement >Reporter: Holden Karau >Priority: Major > > It would be good to support arm64, this is probably not too painful and can > be done by swapping docker build with docker buildx, but there's often edge > cases where some code changes are needed. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
[jira] [Commented] (YUNIKORN-725) Support arm64
[ https://issues.apache.org/jira/browse/YUNIKORN-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493150#comment-17493150 ] Sebastián Risco commented on YUNIKORN-725: -- Any update on this? I'm running a k3s cluster on Raspberry Pis and it would be great to have arm64 images of YuniKorn. > Support arm64 > - > > Key: YUNIKORN-725 > URL: https://issues.apache.org/jira/browse/YUNIKORN-725 > Project: Apache YuniKorn > Issue Type: Improvement >Reporter: Holden Karau >Priority: Major > > It would be good to support arm64, this is probably not too painful and can > be done by swapping docker build with docker buildx, but there's often edge > cases where some code changes are needed. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
