Re: [VOTE] Release Bigtop version 1.4.0
Hey Olaf. Thanks for the test. I’ll put up RC1 to fix the issues Gupdong brought up. Olaf Flebbe 於 2019年5月17日 週五，下午5:26寫道： > Hi Evans > > i am running out of time:tested amd64 dockerhadoop on linux and it run > very smooth! It failed on docker4mac, as we discussed it before: it is an > issue with docker4mac filesystem implementation > > +1 from my side > > we should update the KEYS file since it contains some invalid keys. > > Olaf > > > > Am 14.05.2019 um 14:37 schrieb Evans Ye : > > > > Hi Guodong, > > > > Thanks for the feedback. You're right the images should be 1.4.0 instead > of > > trunk. Let me fix this and have RC1 out. > > I'll add notes in our release guide before we implemented something to > > better handle this automatically. > > > > For missing trunk images, I've uploaded. > > > > Evans > > > > Guodong Xu 於 2019年5月14日 週二 上午11:29寫道： > > > >> Hi, Evans > >> > >> I checked out tag release-1.4.0-RC0. > >> > >> Two issues: > >> 1. for 1.4.0 release, should we change config yaml files to fetch from > >> 'trunk' to '1.4.0'? Like this: > >> $ git diff > >> diff --git a/provisioner/docker/config_centos-7.yaml > >> b/provisioner/docker/config_centos-7.yaml > >> index ab0e9d8..916bf0e 100644 > >> --- a/provisioner/docker/config_centos-7.yaml > >> +++ b/provisioner/docker/config_centos-7.yaml > >> @@ -15,7 +15,7 @@ > >> > >> docker: > >> memory_limit: "4g" > >> -image: "bigtop/puppet:trunk-centos-7" > >> +image: "bigtop/puppet:1.4.0-centos-7" > >> > >> repo: "http://repos.bigtop.apache.org/releases/1.4.0/centos/7/$basearch > " > >> distro: centos > >> > >> > >> 2. for bigtop/puppet images in docker.hub, it misses > >> 'bigtop/puppet:trunk-centos-7-aarch64'. See the error message below when > >> you run ./docker-hadoop.sh for arm. Can you or somebody upload > >> 'bigtop/puppet:trunk-centos-7-aarch64' to docker hub? > >> > >> $ ./docker-hadoop.sh -C config_centos-7.yaml -c 3 > >> Environment check... > >> Check docker: > >> Docker version 18.09.2, build 6247962 > >> Check docker-compose: > >> docker-compose version 1.18.0, build 8dd22a9 > >> Check ruby: > >> ruby 2.0.0p648 (2015-12-16) [aarch64-linux] > >> Pulling bigtop (bigtop/puppet:trunk-centos-7-aarch64)... > >> ERROR: manifest for bigtop/puppet:trunk-centos-7-aarch64 not found > >> > >> [LOG] Docker container(s) startup failed! > >> > >> -Guodong Xu > >> > >>> On Fri, May 10, 2019 at 6:49 PM Evans Ye wrote: > >>> > >>> Let me try to build up those artifacts and see how things go ;) > >>> > >>> Olaf Flebbe 於 2019年5月10日 週五 下午1:01寫道： > >>> > Hi Evans, > > debian 9 should be supported on ppc64le > > olaf > > Von meinem iPad gesendet > > > Am 10.05.2019 um 03:27 schrieb Evans Ye : > > > > Yes it's expected. The failure was cause by not having corresponding > > images. And all the failed builds are not supported in bigtop: > > ARM: opensuse 42.3 > > PPC: centos 7, debian 9, opensuse 42.3 > > > > Youngwoo Kim (김영우) 於 2019年5月10日 週五 上午8:40寫道： > > > >> There are build failures on our CI, aarch64 and ppc64le. Is this > expected > >> results? > >> > >> - Youngwoo > >> > >>> On Mon, May 6, 2019 at 1:33 AM Evans Ye > >> wrote: > >>> > >>> Hi folks, > >>> > >>> This is the vote for release 1.4.0 of Apache Bigtop. > >>> > >>> It fixes the following issues: > >>> * > >>> > >>> > >> > > >>> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344113&projectId=12311420 > >>> > >>> The vote will be going for at least 168 hours and will be closed on > >> Sunday, > >>> *May 12th, 2019 at noon PDT*. Please download, test and vote with > >>> > >>> [ ] +1, accept RC0 as the official 1.4.0 release of Apache Bigtop > >>> [ ] +0, I don't care either way, > >>> [ ] -1, do not accept RC0 as the official 1.4.0 release of Apache > Bigtop, > >>> because... > >>> > >>> Source and binary files: > >>> * https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.4.0-RC0 > >>> > >>> Maven staging repo: > >>> * > >> > >>> > https://repository.apache.org/content/repositories/orgapachebigtop-1022 > >>> > >>> The git tag to be voted upon is release-1.4.0-RC0 > >>> > >>> Bigtop's KEYS file containing PGP keys we use to sign the release: > >>> * https://dist.apache.org/repos/dist/release/bigtop/KEYS > >>> > >>> Bigtop 1.4.0 CI result: > >>> * Packaging: > https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0 > >>> * Deploy & Test: > >>> > >> > > >>> > >> > https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0 > >>> > >> > > >>> > >> >
Re: [DISCUSS] RFC for adding a new Ambari Mpack for Bigtop
Thanks for clarifying. Overall sounds good to me! +1 If no objection, you can take this as a silence agreement from other community members. Yuqi Gu 於 2019年5月17日 週五，下午5:53寫道： > Hi Evans, > > Really thanks for your comments! > > >>* What kind of change do you expect in patch3-AMBARI-bigtop-mpack.diff?* > > > > * Is it served like a mpack selector to override the default > mpack(probabalyCDH/HDP) released by Ambari?A general idea is we > tend not to add patches in Bigtop. A better way is to patch in upstream, > unless it's more reasonable to reside in Bigtop.* > I'm not sure whether Ambari guys will show interests in Bigtop-mpack or > not, so it may take long time to put the patch in Ambari upstream. > 'patch3-AMBARI-bigtop-mpack.diff' is the temporary copy of the > 'Bigtop-Mpack' patch which would be submitted to Ambari upstream. > It also can be used to add the new feature of Bigtop-Mpack in Bigtop 1.4.0 > release or in 1.5.0 release. > When 'Bigtop-mpack' is merged by Ambari upstream, we will remove it from > Bigtop. > > >> *I think we can just add a task in package.gradle that will tar all the* > * required scripts for Bigtop mpack. Simple is better.* > Yeah, it makes sense. > > BRs, > Yuqi > > > > > > > On Tue, 14 May 2019 at 19:52, Evans Ye wrote: > > > Hey Yugi, > > > > Thanks for bringing up the discussion. Overall I think the proposal looks > > good. > > Questions though, > > > > | 3. Put the Bigtop-Mpack patch in 'bigtop-packages/src/common/ambari' as > > 'patch3-AMBARI-bigtop-mpack.diff'. > > What kind of change do you expect in patch3-AMBARI-bigtop-mpack.diff? > > Is it served like a mpack selector to override the default > mpack(probabaly > > CDH/HDP) released by Ambari? > > A general idea is we tend not to add patches in Bigtop. A better way is > to > > patch in upstream, unless it's more reasonable to reside in Bigtop. > > > > | 4. Modify the Bigtop scripts related for releasing Bigtop-mpack tarball > > for users > > I think we can just add a task in package.gradle that will tar all the > > required scripts for Bigtop mpack. Simple is better. > > > > For other's who'd like to know more about the RFC: > > https://issues.apache.org/jira/browse/BIGTOP-3211 > > > > Best, > > Evans > > > > > > > > Yuqi Gu 於 2019年5月14日 週二 下午7:22寫道： > > > > > Hi folks, > > > > > > I'd like to propose a new Ambari Mpack for Bigtop, which is described > in > > > BIGTOP-3211. > > > (Also see in: RFCs for Bigtop-mpack > > > < > > > > > > https://issues.apache.org/jira/secure/attachment/12967363/RFCs%20for%20Bigtop-mpack.pdf > > > > > > > ) > > > > > > Ambari is a Apache project to make Hadoop management simpler. > > > Bigtop Mpack uses Ambari management pack to add management capability > for > > > Bigtop components packages. > > > It can significantly improve the efficiency on deployment and > management > > of > > > Bigtop components and give convenience to > > > the Bigtop users who'd like to adopt Bigtop-Ambari to deploy and > manager > > > other Bigtop components such as Hadoop, Spark,Hive, etc. > > > > > > How-to-achieve-it: > > > 1. Contribute the Bigtop-Mpack to Apache Ambari based on its latest > > release > > > 2.7.3. > > > 2. Upgrade the Ambari from 2.6.1 to 2.7.3 in Bigtop. > > > 3. Put the Bigtop-Mpack patch in 'bigtop-packages/src/common/ambari' as > > > 'patch3-AMBARI-bigtop-mpack.diff'. > > > 4. Modify the Bigtop scripts related for releasing Bigtop-mpack tarball > > for > > > users > > > > > > Any comments are appreciated and welcome! > > > Thanks! > > > > > > BRs, > > > Yuqi > > > > > >
Re: [DISCUSS] RFC for adding a new Ambari Mpack for Bigtop
Hi Evans, Really thanks for your comments! >>* What kind of change do you expect in patch3-AMBARI-bigtop-mpack.diff?* * Is it served like a mpack selector to override the default mpack(probabalyCDH/HDP) released by Ambari?A general idea is we tend not to add patches in Bigtop. A better way is to patch in upstream, unless it's more reasonable to reside in Bigtop.* I'm not sure whether Ambari guys will show interests in Bigtop-mpack or not, so it may take long time to put the patch in Ambari upstream. 'patch3-AMBARI-bigtop-mpack.diff' is the temporary copy of the 'Bigtop-Mpack' patch which would be submitted to Ambari upstream. It also can be used to add the new feature of Bigtop-Mpack in Bigtop 1.4.0 release or in 1.5.0 release. When 'Bigtop-mpack' is merged by Ambari upstream, we will remove it from Bigtop. >> *I think we can just add a task in package.gradle that will tar all the* * required scripts for Bigtop mpack. Simple is better.* Yeah, it makes sense. BRs, Yuqi On Tue, 14 May 2019 at 19:52, Evans Ye wrote: > Hey Yugi, > > Thanks for bringing up the discussion. Overall I think the proposal looks > good. > Questions though, > > | 3. Put the Bigtop-Mpack patch in 'bigtop-packages/src/common/ambari' as > 'patch3-AMBARI-bigtop-mpack.diff'. > What kind of change do you expect in patch3-AMBARI-bigtop-mpack.diff? > Is it served like a mpack selector to override the default mpack(probabaly > CDH/HDP) released by Ambari? > A general idea is we tend not to add patches in Bigtop. A better way is to > patch in upstream, unless it's more reasonable to reside in Bigtop. > > | 4. Modify the Bigtop scripts related for releasing Bigtop-mpack tarball > for users > I think we can just add a task in package.gradle that will tar all the > required scripts for Bigtop mpack. Simple is better. > > For other's who'd like to know more about the RFC: > https://issues.apache.org/jira/browse/BIGTOP-3211 > > Best, > Evans > > > > Yuqi Gu 於 2019年5月14日 週二 下午7:22寫道： > > > Hi folks, > > > > I'd like to propose a new Ambari Mpack for Bigtop, which is described in > > BIGTOP-3211. > > (Also see in: RFCs for Bigtop-mpack > > < > > > https://issues.apache.org/jira/secure/attachment/12967363/RFCs%20for%20Bigtop-mpack.pdf > > > > > ) > > > > Ambari is a Apache project to make Hadoop management simpler. > > Bigtop Mpack uses Ambari management pack to add management capability for > > Bigtop components packages. > > It can significantly improve the efficiency on deployment and management > of > > Bigtop components and give convenience to > > the Bigtop users who'd like to adopt Bigtop-Ambari to deploy and manager > > other Bigtop components such as Hadoop, Spark,Hive, etc. > > > > How-to-achieve-it: > > 1. Contribute the Bigtop-Mpack to Apache Ambari based on its latest > release > > 2.7.3. > > 2. Upgrade the Ambari from 2.6.1 to 2.7.3 in Bigtop. > > 3. Put the Bigtop-Mpack patch in 'bigtop-packages/src/common/ambari' as > > 'patch3-AMBARI-bigtop-mpack.diff'. > > 4. Modify the Bigtop scripts related for releasing Bigtop-mpack tarball > for > > users > > > > Any comments are appreciated and welcome! > > Thanks! > > > > BRs, > > Yuqi > > >
Re: [VOTE] Release Bigtop version 1.4.0
Hi Evans i am running out of time:tested amd64 dockerhadoop on linux and it run very smooth! It failed on docker4mac, as we discussed it before: it is an issue with docker4mac filesystem implementation +1 from my side we should update the KEYS file since it contains some invalid keys. Olaf > Am 14.05.2019 um 14:37 schrieb Evans Ye : > > Hi Guodong, > > Thanks for the feedback. You're right the images should be 1.4.0 instead of > trunk. Let me fix this and have RC1 out. > I'll add notes in our release guide before we implemented something to > better handle this automatically. > > For missing trunk images, I've uploaded. > > Evans > > Guodong Xu 於 2019年5月14日 週二 上午11:29寫道： > >> Hi, Evans >> >> I checked out tag release-1.4.0-RC0. >> >> Two issues: >> 1. for 1.4.0 release, should we change config yaml files to fetch from >> 'trunk' to '1.4.0'? Like this: >> $ git diff >> diff --git a/provisioner/docker/config_centos-7.yaml >> b/provisioner/docker/config_centos-7.yaml >> index ab0e9d8..916bf0e 100644 >> --- a/provisioner/docker/config_centos-7.yaml >> +++ b/provisioner/docker/config_centos-7.yaml >> @@ -15,7 +15,7 @@ >> >> docker: >> memory_limit: "4g" >> -image: "bigtop/puppet:trunk-centos-7" >> +image: "bigtop/puppet:1.4.0-centos-7" >> >> repo: "http://repos.bigtop.apache.org/releases/1.4.0/centos/7/$basearch"; >> distro: centos >> >> >> 2. for bigtop/puppet images in docker.hub, it misses >> 'bigtop/puppet:trunk-centos-7-aarch64'. See the error message below when >> you run ./docker-hadoop.sh for arm. Can you or somebody upload >> 'bigtop/puppet:trunk-centos-7-aarch64' to docker hub? >> >> $ ./docker-hadoop.sh -C config_centos-7.yaml -c 3 >> Environment check... >> Check docker: >> Docker version 18.09.2, build 6247962 >> Check docker-compose: >> docker-compose version 1.18.0, build 8dd22a9 >> Check ruby: >> ruby 2.0.0p648 (2015-12-16) [aarch64-linux] >> Pulling bigtop (bigtop/puppet:trunk-centos-7-aarch64)... >> ERROR: manifest for bigtop/puppet:trunk-centos-7-aarch64 not found >> >> [LOG] Docker container(s) startup failed! >> >> -Guodong Xu >> >>> On Fri, May 10, 2019 at 6:49 PM Evans Ye wrote: >>> >>> Let me try to build up those artifacts and see how things go ;) >>> >>> Olaf Flebbe 於 2019年5月10日 週五 下午1:01寫道： >>> Hi Evans, debian 9 should be supported on ppc64le olaf Von meinem iPad gesendet > Am 10.05.2019 um 03:27 schrieb Evans Ye : > > Yes it's expected. The failure was cause by not having corresponding > images. And all the failed builds are not supported in bigtop: > ARM: opensuse 42.3 > PPC: centos 7, debian 9, opensuse 42.3 > > Youngwoo Kim (김영우) 於 2019年5月10日 週五 上午8:40寫道： > >> There are build failures on our CI, aarch64 and ppc64le. Is this expected >> results? >> >> - Youngwoo >> >>> On Mon, May 6, 2019 at 1:33 AM Evans Ye >> wrote: >>> >>> Hi folks, >>> >>> This is the vote for release 1.4.0 of Apache Bigtop. >>> >>> It fixes the following issues: >>> * >>> >>> >> >>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344113&projectId=12311420 >>> >>> The vote will be going for at least 168 hours and will be closed on >> Sunday, >>> *May 12th, 2019 at noon PDT*. Please download, test and vote with >>> >>> [ ] +1, accept RC0 as the official 1.4.0 release of Apache Bigtop >>> [ ] +0, I don't care either way, >>> [ ] -1, do not accept RC0 as the official 1.4.0 release of Apache Bigtop, >>> because... >>> >>> Source and binary files: >>> * https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.4.0-RC0 >>> >>> Maven staging repo: >>> * >> >>> https://repository.apache.org/content/repositories/orgapachebigtop-1022 >>> >>> The git tag to be voted upon is release-1.4.0-RC0 >>> >>> Bigtop's KEYS file containing PGP keys we use to sign the release: >>> * https://dist.apache.org/repos/dist/release/bigtop/KEYS >>> >>> Bigtop 1.4.0 CI result: >>> * Packaging: https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0 >>> * Deploy & Test: >>> >> >>> >> https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0 >>> >> >>> >>
