Hi Jason, The command you executed (./gradlew hadoop-pkg-ind, without any option) downloads the "trunk" Docker image of CentOS 7 [1] for building. That image is based on the master branch.
On master, we've upgraded Hadoop to 3.x, which requires CMake 3.x to be built. CMake 3 is provided as the "cmake3" package on CentOS 7, so we changed the package dependency for Hadoop in BIGTOP-3472 [2]. Therefore, the trunk image of CentOS 7 has the "cmake3" package installed, instead of "cmake". On the other hand, on the 1.5.0 branch you're working, building Hadoop rpm still requires the "cmake" package regardless of distro [3]. This inconsistency caused your build failure. So would you try `./gradlew hadoop-pkg-ind` with a `-Pprefix=1.5.0` option? It will download the "1.5.0" CentOS 7 image and is supposed to work with your working branch. :) [1]: https://hub.docker.com/layers/bigtop/slaves/trunk-centos-7/images/sha256-8f5f09663908ac2fce7df11203f1613026cdcd7f52bda7efe608b3f8e9fba8b8 [2]: https://github.com/apache/bigtop/commit/e952272ed7aa28d7e2c36b26d81093ca01aa46a8 [3]: https://github.com/apache/bigtop/blob/release-1.5.0/bigtop-packages/src/rpm/hadoop/SPECS/hadoop.spec#L188 Kengo Seki <[email protected]> On Sat, Feb 13, 2021 at 9:54 AM Jason Wen <[email protected]> wrote: > > Hi Matt, > > Thanks for help. > This is the result of `ls` of the extracted dir: > ADVISEADDBEAR:bigtop-1.5.0 zhenshan.wen$ ls > BUILDING.txt NOTICE bigtop-data-generators > bigtop-tests build.gradle gradlew > provisioner test > CHANGES.txt README.md bigtop-deploy > bigtop.bom docker output > release.gradle > LICENSE bigtop-bigpetstore bigtop-packages > bigtop_toolchain docs packages.gradle > settings.gradle > MAINTAINERS.txt bigtop-ci bigtop-test-framework build > gradle pom.xml src > > And this is the command that I used > ./gradlew hadoop-pkg-ind > > I am running this on my mac. > > Thanks, > Jason > From: Matt Andruff <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Friday, February 12, 2021 at 2:07 PM > To: "[email protected]" <[email protected]> > Subject: Re: Bigtop 1.5 build hadoop package error > > so I did finish the build of 1.5 without issues.... but I'm ready to help > figure out why your build didn't work. > > Let me know about those asks I made. > > On Fri, Feb 12, 2021 at 4:26 PM Matt Andruff > <[email protected]<mailto:[email protected]>> wrote: > Hmm, > > I setup ec2 instance, and (./gradlew hadoop-pkg-ind) is humming away... been > building for 7mins.... I'll see if it finishes but appears the docker image > is working as expected. > > By chance where you running the command as 'root' ? > > On Fri, Feb 12, 2021 at 4:04 PM Matt Andruff > <[email protected]<mailto:[email protected]>> wrote: > Hey Jason, > > I'm just spinning up an amazon box to check it out. > > Can you supply: > the exact command you used (include the user) > `ls` of the extracted dir. > > Thanks > > Matt > > On Fri, Feb 12, 2021 at 3:31 PM Jason Wen <[email protected]> > wrote: > I think when I issue './gradlew hadoop-pkg-ind' I am using the pre-built > docker image to build Hadoop packages. > If that is the case, shouldn't the docker image have the necessary tools > installed like cmake? > > Thanks, > Jason > > On 2/12/21, 12:27 PM, "Konstantin Boudnik" > <[email protected]<mailto:[email protected]>> wrote: > > I think here's the reason > > error: Failed build dependencies:Installing > /bigtop/output/hadoop/hadoop-2.10.1-1.el7.src.rpm > > cmake is needed by hadoop-2.10.1-1.el7.x86_64 > > You need to have a set of tools to be able to build the stack. We are > doing thing through setting up the toolchain or using a specially > crafted container, that includes all the toolchain components. > > Check this out > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_BIGTOP_How-2Bto-2Bbuild-2BBigtop-2Dtrunk&d=DwIDaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=Y6BHLw6qDSERpiZ37SZEx9Xeqi8kGIrO9FpCUidwlCk&s=JRGSKpSGqaSNjFM4vw2Q3_J7_QeEMq3akC8eJ_VTZnM&e= > > -- > With regards, > Cos > > On 12.02.2021 23:20, Jason Wen wrote: > > Hi, > > > > I am new to bigtop community. I downloaded bigtop 1.5 release and run > ‘./gradlew hadoop-pkg-ind` in the extract dir. But the build fails with the > following error message. The documentation that I followed is > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_BIGTOP_Quickstart-2BGuide-253A-2BBigtop-2BIntegration-2BTest-2BFramework-2B2.0&d=DwIDaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=Y6BHLw6qDSERpiZ37SZEx9Xeqi8kGIrO9FpCUidwlCk&s=TNLFDA7MZnM2ty9fVqJHhrSbl3Ix9DFuwsauzCMVYME&e= > > > > Can anyone shed light how to solve the build problem? > > > >> Task :hadoop-srpm > > Caching disabled for task ':hadoop-srpm' because: > > Build cache is disabled > > Task ':hadoop-srpm' is not up-to-date because: > > Task has not declared any outputs despite executing actions. > > Nothing to do. Exiting... > > :hadoop-srpm (Thread[Execution worker for ':',5,main]) completed. Took > 0.001 secs. > > :hadoop-rpm (Thread[Execution worker for ':',5,main]) started. > > > >> Task :hadoop-rpm FAILED > > Caching disabled for task ':hadoop-rpm' because: > > Build cache is disabled > > Task ':hadoop-rpm' is not up-to-date because: > > Task has not declared any outputs despite executing actions. > > Starting process 'command 'rpmbuild''. Working directory: /bigtop > Command: rpmbuild --define _topdir /bigtop/build/hadoop/rpm/ --define > hadoop_base_version 2.10.1 --define hadoop_version 2.10.1 --define > hadoop_version 2.10.1 --define hadoop_release 1%{?dist} --rebuild > /bigtop/output/hadoop/hadoop-2.10.1-1.el7.src.rpm --define do_maven_deploy > false --define maven_deploy_source false > > Successfully started process 'command 'rpmbuild'' > > error: Failed build dependencies:Installing > /bigtop/output/hadoop/hadoop-2.10.1-1.el7.src.rpm > > > > cmake is needed by hadoop-2.10.1-1.el7.x86_64 > > > > :hadoop-rpm (Thread[Execution worker for ':',5,main]) completed. Took > 0.628 secs. > > FAILURE: Build failed with an exception. > > > > * Where: > > Script '/bigtop/packages.gradle' line: 507 > > > > * What went wrong: > > Execution failed for task ':hadoop-rpm'. > >> Process 'command 'rpmbuild'' finished with non-zero exit value 1 > > > > * Try: > > Run with --stacktrace option to get the stack trace. Run with --debug > option to get more log output. Run with --scan to get full insights. > > > > * Get more help at > https://urldefense.proofpoint.com/v2/url?u=https-3A__help.gradle.org&d=DwIDaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=Y6BHLw6qDSERpiZ37SZEx9Xeqi8kGIrO9FpCUidwlCk&s=8r0X9nES1diOG-8Y7uYlAa8-w1oXd4yvV0EpCwxN-e8&e= > > > > BUILD FAILED in 39s > > 5 actionable tasks: 5 executed > > + RESULT=1 > > + mkdir -p output > > + docker cp > 89a442c078b2c620285f46b9661c6e1d96df73365b7b2ad8200644dd995c317b:/bigtop/build > . > > + docker cp > 89a442c078b2c620285f46b9661c6e1d96df73365b7b2ad8200644dd995c317b:/bigtop/output > . > > + docker rm -f > 89a442c078b2c620285f46b9661c6e1d96df73365b7b2ad8200644dd995c317b > > 89a442c078b2c620285f46b9661c6e1d96df73365b7b2ad8200644dd995c317b > > + '[' 1 -ne 0 ']' > > + exit 1 > > + docker rm -f > 89a442c078b2c620285f46b9661c6e1d96df73365b7b2ad8200644dd995c317b > > Error: No such container: > 89a442c078b2c620285f46b9661c6e1d96df73365b7b2ad8200644dd995c317b > > > >> Task :hadoop-pkg-ind FAILED > > > > FAILURE: Build failed with an exception. > > > > * Where: > > Script '/Users/zhenshan.wen/code/learning/bigtop-1.5.0/packages.gradle' > line: 675 > > > > * What went wrong: > > Execution failed for task ':hadoop-pkg-ind'. > >> Process 'command 'bash'' finished with non-zero exit value 1 > > > > * Try: > > Run with --stacktrace option to get the stack trace. Run with --info or > --debug option to get more log output. Run with --scan to get full insights. > > > > * Get more help at > https://urldefense.proofpoint.com/v2/url?u=https-3A__help.gradle.org&d=DwIDaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=Y6BHLw6qDSERpiZ37SZEx9Xeqi8kGIrO9FpCUidwlCk&s=8r0X9nES1diOG-8Y7uYlAa8-w1oXd4yvV0EpCwxN-e8&e= > > > > BUILD FAILED in 1m 7s > > 1 actionable task: 1 executed > > > > Thanks, > > Jason > >
