Re: Create multiarch docker images for bigtop-puppet/slaves
Thanks for pointing the resources and suggestions , Roman, :) Regards the multi-arch, my intension is to simplify the images name, so the actual underlying things (arch-depent images) are transparent to Bigtop users/developers. Say, they can just issue same cmd: docker pull bigtop/puppet:1.5-centos-7, on any platform, and then start doing coll stuffs. And another puprpose is to simplify provision/test process. As you can see there are some arch related yaml files in https://github.com/apache/bigtop/tree/master/provisioner/docker. If multi-arch is enabled, we can remove those config-*-aarch64.yaml files, and use just one unified yaml file for certain distro/packages. Also I think this could help devOps guys if they're using Bigtop to manage deployment. 2018-05-29 14:04 GMT+08:00 Roman Shaposhnik : > Here's a few points from the peanut gallery: in general I find > Docker multiarch pretty useful, but at the same time its not like > having a schema like: > bigtop/:-- version>- > wouldn't achieve the same. I'd say go with what feels simpler. > > As for scripts that create those -- that's just puppet in: > https://github.com/apache/bigtop/tree/master/bigtop_toolchain > > So your Dockerfile is super-straightforward -- start with the base, > add puppet via: >https://github.com/apache/bigtop/blob/master/bigtop_ > toolchain/bin/puppetize.sh > and run the puppet. > > In fact, I believe that this bakes the images (unless it bitrotted > somehow): >https://github.com/apache/bigtop/blob/master/docker/ > bigtop-slaves/build.sh > > > Thanks, > Roman. > > On Sun, May 27, 2018 at 7:55 PM, Jun HE wrote: > > Thanks for comments, Evans. > > > > There should not be much efforts to enable multi-arch. The question is > how > > we plan to support it in Bigtop. > > > > I can think of several items to discuss for this feature: > > 1. Docker images name conventions: > > For this, I'd suggest to using this: > > * for actual underlying docker image running on specific platform, > name > > it as: > > bigtop/:-- version>-, > > say: bigtop/puppet:trunk-debian-9-amd64, > > bigtop/slaves:1.3.0-fedora-26-aarch64 > > * for virtual image, name it as: bigtop/: > > --, > > say: bigtop/puppet:trunk-debian-9, bigtop/slaves:1.3.0-fedora-26 > > 2. How to create these images/manifests > > I didn't find any cmds/scripts in Bigtop source to do the docker hub > > uploading work. So I guess we're doing it manually for now. If we're > going > > to use multi-arch, we need to upload the virtual image's manifest file to > > docker hub besides those arch dependent images. > > Docker cli has a cmd to do such work: docker manifest. But it is not > in > > distros' docker releases. And it is marked as "experimental" feature, > user > > needs to change docker settings to use it. See > > https://docs.docker.com/edge/engine/reference/commandline/ > manifest/#description. > > I created a shell script version ( > > https://github.com/JunHe77/docker-manifest-sh) to do the same thing, use > > same cmd syntax. Hope this can help. > > 3. Update existing settings/yaml files to use the virutal distro image. > > Once the images/manifests are ready in docker hub, we can update > > existing settings/yaml files to use "virtual" bigtop images. This should > > take too much efforts. And we'll get unified, beautiful code for > different > > archs. > > > > Pls raise your thoughts, questions, and etc. Once we settle these > > workflows/specs down, I'm sure it will be ready before Sept. > > > > Thanks. > > > > 2018-05-28 1:45 GMT+08:00 Evans Ye : > > > >> I'd say this is a great idea. > >> The only reason we haven't adopt multi-arch is because we started to > adopt > >> docker at early stage so there's no such feature then. > >> > >> Jun can you roughly estimate the effort? If we're going for it, we > better > >> have it ready in 1.3.0 before ApacheCon which is around September. > >> > >> > >> 2018-05-17 17:19 GMT+08:00 Naresh Bhat : > >> > >> > On 17 May 2018 at 13:21, Jun HE wrote: > >> > > Hi, guys, > >> > > > >> > > As bigtop has simplified the way to create docker images for > distros, > >> by > >> > > BIGTOP-2922, I'm thinking of multi-arch docker images support to > ease > >> > > building/test/development. > >> > > > >> > > Multi-arch feature has been supported since 2017, you may check this > >> > link: > >> > > https://blog.docker.com/2017/11/multi-arch-all-the-things/ > >> > > In simple words, multi-arch is a virtual image which contains > several > >> > > architecture dependent images. When user try to pull this virtual > image > >> > > dockhub will send back the actual image based on the architecture > he is > >> > > using. > >> > > > >> > > Use multi-arch will help: > >> > > * development: user don't have to issue different cmd to pull > >> > puppet/slaves > >> > > images, "docker pull bigtop/slaves:trunk-debian-9" is enough > instead of > >> > "docker > >> > > pull bigtop/slaves:trunk-debian-9-ppc64le" or "docker pull > >>
Re: Create multiarch docker images for bigtop-puppet/slaves
Here's a few points from the peanut gallery: in general I find Docker multiarch pretty useful, but at the same time its not like having a schema like: bigtop/:--- wouldn't achieve the same. I'd say go with what feels simpler. As for scripts that create those -- that's just puppet in: https://github.com/apache/bigtop/tree/master/bigtop_toolchain So your Dockerfile is super-straightforward -- start with the base, add puppet via: https://github.com/apache/bigtop/blob/master/bigtop_toolchain/bin/puppetize.sh and run the puppet. In fact, I believe that this bakes the images (unless it bitrotted somehow): https://github.com/apache/bigtop/blob/master/docker/bigtop-slaves/build.sh Thanks, Roman. On Sun, May 27, 2018 at 7:55 PM, Jun HE wrote: > Thanks for comments, Evans. > > There should not be much efforts to enable multi-arch. The question is how > we plan to support it in Bigtop. > > I can think of several items to discuss for this feature: > 1. Docker images name conventions: > For this, I'd suggest to using this: > * for actual underlying docker image running on specific platform, name > it as: > bigtop/:---, > say: bigtop/puppet:trunk-debian-9-amd64, > bigtop/slaves:1.3.0-fedora-26-aarch64 > * for virtual image, name it as: bigtop/: > --, > say: bigtop/puppet:trunk-debian-9, bigtop/slaves:1.3.0-fedora-26 > 2. How to create these images/manifests > I didn't find any cmds/scripts in Bigtop source to do the docker hub > uploading work. So I guess we're doing it manually for now. If we're going > to use multi-arch, we need to upload the virtual image's manifest file to > docker hub besides those arch dependent images. > Docker cli has a cmd to do such work: docker manifest. But it is not in > distros' docker releases. And it is marked as "experimental" feature, user > needs to change docker settings to use it. See > https://docs.docker.com/edge/engine/reference/commandline/manifest/#description. > I created a shell script version ( > https://github.com/JunHe77/docker-manifest-sh) to do the same thing, use > same cmd syntax. Hope this can help. > 3. Update existing settings/yaml files to use the virutal distro image. > Once the images/manifests are ready in docker hub, we can update > existing settings/yaml files to use "virtual" bigtop images. This should > take too much efforts. And we'll get unified, beautiful code for different > archs. > > Pls raise your thoughts, questions, and etc. Once we settle these > workflows/specs down, I'm sure it will be ready before Sept. > > Thanks. > > 2018-05-28 1:45 GMT+08:00 Evans Ye : > >> I'd say this is a great idea. >> The only reason we haven't adopt multi-arch is because we started to adopt >> docker at early stage so there's no such feature then. >> >> Jun can you roughly estimate the effort? If we're going for it, we better >> have it ready in 1.3.0 before ApacheCon which is around September. >> >> >> 2018-05-17 17:19 GMT+08:00 Naresh Bhat : >> >> > On 17 May 2018 at 13:21, Jun HE wrote: >> > > Hi, guys, >> > > >> > > As bigtop has simplified the way to create docker images for distros, >> by >> > > BIGTOP-2922, I'm thinking of multi-arch docker images support to ease >> > > building/test/development. >> > > >> > > Multi-arch feature has been supported since 2017, you may check this >> > link: >> > > https://blog.docker.com/2017/11/multi-arch-all-the-things/ >> > > In simple words, multi-arch is a virtual image which contains several >> > > architecture dependent images. When user try to pull this virtual image >> > > dockhub will send back the actual image based on the architecture he is >> > > using. >> > > >> > > Use multi-arch will help: >> > > * development: user don't have to issue different cmd to pull >> > puppet/slaves >> > > images, "docker pull bigtop/slaves:trunk-debian-9" is enough instead of >> > "docker >> > > pull bigtop/slaves:trunk-debian-9-ppc64le" or "docker pull >> > > bigtop/slaves:trunk-debian-9-aarch64" >> > > * test: unified yaml file for same distro. refer example in >> > > provisioner/docker/config_* >> > > >> > > So how do you think? :) >> > >> > Thanks Jun for initiating the thread. >> > +1 this makes sense. Is there any reason why we have not consider for >> > using multi-arch images ? If anybody explains advantage/disadvantage >> > of using multi-arch docker image in bigtop, it will help us to >> > understand in a better way. >> > >> > > >> > > Regards, >> > > >> > > Jun >> > >>
Re: Create multiarch docker images for bigtop-puppet/slaves
Thanks for comments, Evans. There should not be much efforts to enable multi-arch. The question is how we plan to support it in Bigtop. I can think of several items to discuss for this feature: 1. Docker images name conventions: For this, I'd suggest to using this: * for actual underlying docker image running on specific platform, name it as: bigtop/:---, say: bigtop/puppet:trunk-debian-9-amd64, bigtop/slaves:1.3.0-fedora-26-aarch64 * for virtual image, name it as: bigtop/: --, say: bigtop/puppet:trunk-debian-9, bigtop/slaves:1.3.0-fedora-26 2. How to create these images/manifests I didn't find any cmds/scripts in Bigtop source to do the docker hub uploading work. So I guess we're doing it manually for now. If we're going to use multi-arch, we need to upload the virtual image's manifest file to docker hub besides those arch dependent images. Docker cli has a cmd to do such work: docker manifest. But it is not in distros' docker releases. And it is marked as "experimental" feature, user needs to change docker settings to use it. See https://docs.docker.com/edge/engine/reference/commandline/manifest/#description. I created a shell script version ( https://github.com/JunHe77/docker-manifest-sh) to do the same thing, use same cmd syntax. Hope this can help. 3. Update existing settings/yaml files to use the virutal distro image. Once the images/manifests are ready in docker hub, we can update existing settings/yaml files to use "virtual" bigtop images. This should take too much efforts. And we'll get unified, beautiful code for different archs. Pls raise your thoughts, questions, and etc. Once we settle these workflows/specs down, I'm sure it will be ready before Sept. Thanks. 2018-05-28 1:45 GMT+08:00 Evans Ye : > I'd say this is a great idea. > The only reason we haven't adopt multi-arch is because we started to adopt > docker at early stage so there's no such feature then. > > Jun can you roughly estimate the effort? If we're going for it, we better > have it ready in 1.3.0 before ApacheCon which is around September. > > > 2018-05-17 17:19 GMT+08:00 Naresh Bhat : > > > On 17 May 2018 at 13:21, Jun HE wrote: > > > Hi, guys, > > > > > > As bigtop has simplified the way to create docker images for distros, > by > > > BIGTOP-2922, I'm thinking of multi-arch docker images support to ease > > > building/test/development. > > > > > > Multi-arch feature has been supported since 2017, you may check this > > link: > > > https://blog.docker.com/2017/11/multi-arch-all-the-things/ > > > In simple words, multi-arch is a virtual image which contains several > > > architecture dependent images. When user try to pull this virtual image > > > dockhub will send back the actual image based on the architecture he is > > > using. > > > > > > Use multi-arch will help: > > > * development: user don't have to issue different cmd to pull > > puppet/slaves > > > images, "docker pull bigtop/slaves:trunk-debian-9" is enough instead of > > "docker > > > pull bigtop/slaves:trunk-debian-9-ppc64le" or "docker pull > > > bigtop/slaves:trunk-debian-9-aarch64" > > > * test: unified yaml file for same distro. refer example in > > > provisioner/docker/config_* > > > > > > So how do you think? :) > > > > Thanks Jun for initiating the thread. > > +1 this makes sense. Is there any reason why we have not consider for > > using multi-arch images ? If anybody explains advantage/disadvantage > > of using multi-arch docker image in bigtop, it will help us to > > understand in a better way. > > > > > > > > Regards, > > > > > > Jun > > >
Re: Create multiarch docker images for bigtop-puppet/slaves
I'd say this is a great idea. The only reason we haven't adopt multi-arch is because we started to adopt docker at early stage so there's no such feature then. Jun can you roughly estimate the effort? If we're going for it, we better have it ready in 1.3.0 before ApacheCon which is around September. 2018-05-17 17:19 GMT+08:00 Naresh Bhat : > On 17 May 2018 at 13:21, Jun HE wrote: > > Hi, guys, > > > > As bigtop has simplified the way to create docker images for distros, by > > BIGTOP-2922, I'm thinking of multi-arch docker images support to ease > > building/test/development. > > > > Multi-arch feature has been supported since 2017, you may check this > link: > > https://blog.docker.com/2017/11/multi-arch-all-the-things/ > > In simple words, multi-arch is a virtual image which contains several > > architecture dependent images. When user try to pull this virtual image > > dockhub will send back the actual image based on the architecture he is > > using. > > > > Use multi-arch will help: > > * development: user don't have to issue different cmd to pull > puppet/slaves > > images, "docker pull bigtop/slaves:trunk-debian-9" is enough instead of > "docker > > pull bigtop/slaves:trunk-debian-9-ppc64le" or "docker pull > > bigtop/slaves:trunk-debian-9-aarch64" > > * test: unified yaml file for same distro. refer example in > > provisioner/docker/config_* > > > > So how do you think? :) > > Thanks Jun for initiating the thread. > +1 this makes sense. Is there any reason why we have not consider for > using multi-arch images ? If anybody explains advantage/disadvantage > of using multi-arch docker image in bigtop, it will help us to > understand in a better way. > > > > > Regards, > > > > Jun >
Re: Create multiarch docker images for bigtop-puppet/slaves
On 17 May 2018 at 13:21, Jun HE wrote: > Hi, guys, > > As bigtop has simplified the way to create docker images for distros, by > BIGTOP-2922, I'm thinking of multi-arch docker images support to ease > building/test/development. > > Multi-arch feature has been supported since 2017, you may check this link: > https://blog.docker.com/2017/11/multi-arch-all-the-things/ > In simple words, multi-arch is a virtual image which contains several > architecture dependent images. When user try to pull this virtual image > dockhub will send back the actual image based on the architecture he is > using. > > Use multi-arch will help: > * development: user don't have to issue different cmd to pull puppet/slaves > images, "docker pull bigtop/slaves:trunk-debian-9" is enough instead of > "docker > pull bigtop/slaves:trunk-debian-9-ppc64le" or "docker pull > bigtop/slaves:trunk-debian-9-aarch64" > * test: unified yaml file for same distro. refer example in > provisioner/docker/config_* > > So how do you think? :) Thanks Jun for initiating the thread. +1 this makes sense. Is there any reason why we have not consider for using multi-arch images ? If anybody explains advantage/disadvantage of using multi-arch docker image in bigtop, it will help us to understand in a better way. > > Regards, > > Jun
