Hi list,

Long story short: I tried to follow the bigtop quickstart guide from the
wiki but failed. Can you help me please?


I would like to understand the differences between relational databases
that I know well and hadoop where I know next to nothing about. To achieve
that, I want to play around with the tools and products the hadoop
ecosystem has to offer.

My goal is to understand how things work together from feeding some
unstructured data to querying it with some end user tool and everything in
between. It is not about volume or high availability on multiple servers or
just establishing a connection to an existing hadoop installation.
For now, as the first step, I just want to make the pipeline work in
general.

So far beginner friendly documentation that provides a full overview is
sparse, or I haven't looked in the right direction yet. But now I found
this project, which - judging from the 1.4.0 release notes - at first
seemed ideal for my purpose:

Deploying Bigtop is easy: grab the repo/list file for your favorite Linux
> distribution:
>   https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.4.0/repos/
> and you'll be running your very own big data cluster in no time!


But still I got stuck. It is not very clear to me what is actually needed
to get hadoop up and running with bigtop. There seem to be big differences
depending on versions. What are the rpm's good for? They seem to be just a
tomcat server? Why is that needed?

My understanding of bigtop from the Quickstart Guide is that I need to
unpack the bigtop project tar and run gradlew on a system that runs docker.
This should get all tools and sources and build all software, connect it
together, and do some integration tests (maybe with the help of the tomcat
server). Right?

I prepared a script and the logs it produces, so you see exactly what I'm
trying to do. Probably you can spot very easily what the problem is?

Best Regards,
Joachim



P.S.: I tried these -POS values with and without the bigtop repo rpms
installed. With more or less the same result:

bigtop@a956005e4481:~> docker image ls
REPOSITORY          TAG                   IMAGE ID            CREATED
      SIZE
bigtop/slaves       trunk-centos-8        ba284fdb2b47        8 hours ago
      3.51GB
bigtop/slaves       trunk-ubuntu-16.04    43c6ff57c166        8 hours ago
      3.76GB
bigtop/slaves       trunk-centos-7        8809f6c53ef8        8 hours ago
      2.72GB
bigtop/slaves       trunk-opensuse-42.3   e7246a926a13        7 months ago
       2.77GB




P.P.S.: Yesterday I had one testrun that went quite a bit farther, but
failed with a java exception during building an rpm in trunk-opensuse-42.3.
But I cannot reproduce that anymore. It fails much earlier now.



My test runs on a bare metal opensuse 15.1 with its docker package
(19.03.11), but the test itself runs in an opensuse 42.3 docker image to
make it easy to reproduce my environment:

docker run --name bigtop-prep -v /var/run/docker.sock:/var/run/docker.sock
-v /usr/bin/docker:/usr/bin/docker -ti opensuse/leap:42.3 /bin/bash
zypper in -y vim less
zypper in -y tar curl unzip rpm-build java-1_8_0-openjdk-devel
sed -i 's/en_US.UTF-8//' /etc/sysconfig/language
groupadd -g `ls -l /var/run/docker.sock | while read x x x g x; do echo $g;
done` docker
useradd -m -G docker bigtop
zypper ar -G
https://artfiles.org/apache.org/bigtop/bigtop-1.4.0/repos/opensuse42.3/bigtop.repo
zypper in -y which bigtop-utils bigtop-jsvc bigtop-tomcat bigtop-groovy
cp -av  /usr/lib/bigtop-tomcat/conf.template /usr/lib/bigtop-tomcat/conf
sed -i '/<\/tomcat-users>/i \ \ <role rolename="manager"/>'
/usr/lib/bigtop-tomcat/conf/tomcat-users.xml
sed -i '/<\/tomcat-users>/i \ \ <role rolename="manager-gui"/>'
/usr/lib/bigtop-tomcat/conf/tomcat-users.xml
sed -i '/<\/tomcat-users>/i \ \ <user username="tomcat" password="my-PW"
roles="tomcat,manager,manager-gui"/>'
/usr/lib/bigtop-tomcat/conf/tomcat-users.xml
sed -i '/<\/tomcat-users>/i \ \ <user username="joachim" password="my-PW"
roles="tomcat,manager,manager-gui"/>'
/usr/lib/bigtop-tomcat/conf/tomcat-users.xml
sed -i '/<\/tomcat-users>/i \ \ <user username="julian" password="his-PW"
roles="tomcat,manager,manager-gui"/>'
/usr/lib/bigtop-tomcat/conf/tomcat-users.xml
/usr/lib/bigtop-tomcat/bin/startup.sh
su - bigtop
curl
https://apache.lauf-forum.at/bigtop/bigtop-1.4.0/bigtop-1.4.0-project.tar.gz
-o bigtop-1.4.0-project.tar.gz
tar xf bigtop-1.4.0-project.tar.gz
ln -sf bigtop-1.4.0 bigtop
cd bigtop
./gradlew --console=plain hadoop-pkg-ind -POS=opensuse-42.3




This is the complete output of the gradle run (other steps seem fine,
tomcat catalina is running: ports are open but makes no difference):

bigtop@a956005e4481:~/bigtop> ./gradlew --console=plain hadoop-pkg-ind
-POS=opensuse-42.3
 % Total    % Received % Xferd  Average Speed   Time    Time     Time
 Current
                                Dload  Upload   Total   Spent    Left  Speed
 0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--
0
100 74.7M  100 74.7M    0     0  5839k      0  0:00:13  0:00:13 --:--:--
5990k
~/.gradle/wrapper/dists/gradle-4.10.3-bin/016e637c7ef47db2d3e632a3936ca351
~/bigtop
~/bigtop
Welcome to Gradle 4.10.3!
Here are the highlights of this release:
- Incremental Java compilation by default
- Periodic Gradle caches cleanup
- Gradle Kotlin DSL 1.0-RC6
- Nested included builds
- SNAPSHOT plugin versions in the `plugins {}` block
For more details see https://docs.gradle.org/4.10.3/release-notes.html
Starting a Gradle Daemon (subsequent builds will be faster)
> Task :hadoop-pkg-ind FAILED
Building trunk hadoop-pkg on opensuse-42.3 in Docker...
+++ dirname ./bigtop-ci/build.sh
++ cd ./bigtop-ci/..
++ pwd
+ BIGTOP_HOME=/home/bigtop/bigtop
+ '[' 6 -eq 0 ']'
+ [[ 6 -gt 0 ]]
+ key=--prefix
+ case $key in
+ PREFIX=trunk
+ shift
+ shift
+ [[ 4 -gt 0 ]]
+ key=--os
+ case $key in
+ OS=opensuse-42.3
+ shift
+ shift
+ [[ 2 -gt 0 ]]
+ key=--target
+ case $key in
+ TARGET=hadoop-pkg
+ shift
+ shift
+ [[ 0 -gt 0 ]]
+ '[' -z x ']'
+ '[' -z x ']'
+ '[' '' == true ']'
+ IMAGE_NAME=bigtop/slaves:trunk-opensuse-42.3
++ uname -m
+ ARCH=x86_64
+ '[' x86_64 '!=' x86_64 ']'
++ docker run -d bigtop/slaves:trunk-opensuse-42.3 /sbin/init
+
CONTAINER_ID=b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be
+ trap 'docker rm -f
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be' EXIT +
docker cp /home/bigtop/bigtop
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be:/bigtop
+ docker cp /home/bigtop/bigtop/bigtop-ci/entrypoint.sh
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be:/bigtop/entrypoint.sh
Error: No such container:path:
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be:/bigtop
+ docker exec
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be bash -c
'chown -R jenkins:jenkins /bigtop'
+ docker exec --user jenkins
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be bash -c
'cd /bigtop && ./entrypoint.sh  hadoop-pkg --info'
bash: line 0: cd: /bigtop: No such file or directory
+ RESULT=1
+ mkdir -p output
+ docker cp
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be:/bigtop/build
.
Error: No such container:path:
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be:/bigtop/build
+ docker cp
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be:/bigtop/output
.
Error: No such container:path:
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be:/bigtop/output
+ docker rm -f
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be
+ '[' 1 -ne 0 ']'
+ exit 1
+ docker rm -f
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be
Error: No such container:
b21178150cd1ebb8640eebe9e24e27cd9e1d65da4bcf0e22e6c4e725514231be
FAILURE: Build failed with an exception.
* Where:
Script '/home/bigtop/bigtop-1.4.0/packages.gradle' line: 657
* What went wrong:
Execution failed for task ':hadoop-pkg-ind'.
> Process 'command 'bash'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or
--debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 30s
1 actionable task: 1 executed

Reply via email to