[jira] [Created] (BIGTOP-3774) Drop Docker Sandbox and Vagrant Provisioner
Evans Ye created BIGTOP-3774: Summary: Drop Docker Sandbox and Vagrant Provisioner Key: BIGTOP-3774 URL: https://issues.apache.org/jira/browse/BIGTOP-3774 Project: Bigtop Issue Type: Improvement Components: provisioner, sandbox Affects Versions: 3.1.1 Reporter: Evans Ye Assignee: Evans Ye Per discussion in dev list[1], this JIRA aims to remove non-active maintained Vagrant Provisioner and Docker Sandbox. [1] https://lists.apache.org/thread/x1wxby7mjortw0crybnnww8lhhorovlq. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (BIGTOP-3355) [Puppet] Make GPG check for repos a configuration and default to true
Evans Ye created BIGTOP-3355: Summary: [Puppet] Make GPG check for repos a configuration and default to true Key: BIGTOP-3355 URL: https://issues.apache.org/jira/browse/BIGTOP-3355 Project: Bigtop Issue Type: Bug Components: deployment Affects Versions: 1.4.0 Reporter: Evans Ye Assignee: Evans Ye Currently Bigtop Puppet deploys repos w/o GPG check. This is insecure and need to be fixed. We can make disable GPG check a configuration and default to true. For testing purpose, one can set it to false to run w/ local repo or the others. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BIGTOP-3345) Failed to run smoke test due to jenkins do not have permission to cleanup workspace
Evans Ye created BIGTOP-3345:
Summary: Failed to run smoke test due to jenkins do not have
permission to cleanup workspace
Key: BIGTOP-3345
URL: https://issues.apache.org/jira/browse/BIGTOP-3345
Project: Bigtop
Issue Type: Bug
Components: ci
Affects Versions: 1.4.0
Reporter: Evans Ye
{code:java}
ERROR: Failed to clean the workspace
jenkins.util.io.CompositeIOException: Unable to delete
'/home/jenkins/workspace/Build-Deploy-Smoke-Test-Pull-Request-All-Distros/OS/centos-7-x86_64-deploy'.
Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
at
jenkins.util.io.PathRemover.forceRemoveDirectoryContents(PathRemover.java:90)
at hudson.Util.deleteContentsRecursive(Util.java:260)
at hudson.Util.deleteContentsRecursive(Util.java:249)
at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:739)
at
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
at
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.nio.file.FileSystemException:
/home/jenkins/workspace/Build-Deploy-Smoke-Test-Pull-Request-All-Distros/OS/centos-7-x86_64-deploy/bigtop-tests/smoke-tests/logger-test-config/build/resources/main/log4j.properties:
Operation not permitted
{code}
Ref:
https://ci.bigtop.apache.org/view/Test/job/Build-Deploy-Smoke-Test-Pull-Request-All-Distros/OS=centos-7-x86_64-deploy/52/consoleFull
Guessing it's because the job error out early before the cleanup command
getting executed:
{code}
./gradlew realclean docker-provisioner-destroy
{code}
Ref: https://issues.apache.org/jira/browse/BIGTOP-3344
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BIGTOP-3344) Check-in Smoke Test CI script to the codebase
Evans Ye created BIGTOP-3344:
Summary: Check-in Smoke Test CI script to the codebase
Key: BIGTOP-3344
URL: https://issues.apache.org/jira/browse/BIGTOP-3344
Project: Bigtop
Issue Type: Sub-task
Components: ci
Affects Versions: 1.4.0
Reporter: Evans Ye
The current smoke test script:
{code:java}
OS_WO_ARCH=`echo ${OS} | awk -F"-" '{ print $1"-"$2}'`
# clean up
case $(arch) in
x86_64) IMAGE_SUFIX="";;
*) IMAGE_SUFIX="-$(arch)";;
esac
docker run --rm -v `pwd`:/ws bigtop/slaves:trunk-${OS_WO_ARCH}${IMAGE_SUFIX}
bash -c "cd /ws;git clean -xdf"
if [ ${COMPONENT} != "custom" ]; then
echo "COMPONENT = ${COMPONENT}"
else
COMPONENT=${CUSTOM_COMPONENT}
fi
if [ ${COMPONENT} != "skip" ]; then
RUN_PACKAGING="allclean ${COMPONENT}-pkg-ind repo-ind"
ENABLE_LOCAL_REPO_PROPERTY="-Penable_local_repo";
fi
if [ ! -z ${STACK} ]; then
RUN_PROVISIONER="docker-provisioner-destroy docker-provisioner"
STACK_PROPERTY="-Pstack=${STACK}";
NUM_INSTANCES_PROPERTY="-Pnum_instances=${NUM_INSTANCES}";
fi
if [ ! -z ${SMOKE_TESTS} ]; then
SMOKE_TESTS_PROPERTY="-Psmoke_tests=${SMOKE_TESTS}";
fi
# Apply patch
curl -L ${PR}.patch | git am
./gradlew ${RUN_PACKAGING} \
-POS=${OS_WO_ARCH} \
-Pprefix=${PREFIX} \
-Pconfig=config_${OS_WO_ARCH}.yaml \
${RUN_PROVISIONER} \
${ENABLE_LOCAL_REPO_PROPERTY} \
${NUM_INSTANCES_PROPERTY} \
${STACK_PROPERTY} \
${SMOKE_TESTS_PROPERTY} \
--info -s;
./gradlew realclean docker-provisioner-destroy
{code}
Jenkins job:
https://ci.bigtop.apache.org/view/Test/job/Build-Deploy-Smoke-Test-Pull-Request-All-Distros/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BIGTOP-3326) Error out when docker image build failed
Evans Ye created BIGTOP-3326: Summary: Error out when docker image build failed Key: BIGTOP-3326 URL: https://issues.apache.org/jira/browse/BIGTOP-3326 Project: Bigtop Issue Type: New Feature Components: docker Reporter: Evans Ye Currently our docker image build script does not detect whether there's an error when building. The downside is when library download failed, there's no visibility on it and the image is still built and shipped. See BIGTOP-3224 as an example(MVN missing). We can: 1. Abort the docker build on non 0 EXEC 2. Post detect the build log and make remediation afterwards. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BIGTOP-3325) Spark LICENCE files should be exempted by RAT check
Evans Ye created BIGTOP-3325: Summary: Spark LICENCE files should be exempted by RAT check Key: BIGTOP-3325 URL: https://issues.apache.org/jira/browse/BIGTOP-3325 Project: Bigtop Issue Type: New Feature Reporter: Evans Ye -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BIGTOP-3324) Wrap CI scripts into Bigtop codebase
Evans Ye created BIGTOP-3324: Summary: Wrap CI scripts into Bigtop codebase Key: BIGTOP-3324 URL: https://issues.apache.org/jira/browse/BIGTOP-3324 Project: Bigtop Issue Type: New Feature Components: ci Reporter: Evans Ye Currently our CI scripts on ci.bigtop.apache.org are pretty complicated. We should maintain it in git and fetch them from our repo when triggering CI jobs instead of writing them on Jenkins. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BIGTOP-3215) Failed to build bigtop/puppet image for OpenSuSE
Evans Ye created BIGTOP-3215:
Summary: Failed to build bigtop/puppet image for OpenSuSE
Key: BIGTOP-3215
URL: https://issues.apache.org/jira/browse/BIGTOP-3215
Project: Bigtop
Issue Type: Sub-task
Components: docker
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
Fix For: 1.4.0
{code}
Started by upstream project "Docker-Puppet-1.4.0" build number 2
originally caused by:
Started by user Evans Ye
Building remotely on docker-slave-arm-5 (ubuntu-16.04-arm64-deploy
fedora-26-aarch64-deploy aarch64-slave debian-9-arm64-deploy centos-7-aarch64
debian-9-aarch64) in workspace
/home/jenkins/workspace/Docker-Puppet-1.4.0/DISTRO/opensuse-42.3/PLATFORM/aarch64-slave
Cloning the remote Git repository
Cloning repository https://gitbox.apache.org/repos/asf/bigtop.git
> git init
> /home/jenkins/workspace/Docker-Puppet-1.4.0/DISTRO/opensuse-42.3/PLATFORM/aarch64-slave
> # timeout=10
Fetching upstream changes from https://gitbox.apache.org/repos/asf/bigtop.git
> git --version # timeout=10
> git fetch --tags --progress https://gitbox.apache.org/repos/asf/bigtop.git
> +refs/heads/*:refs/remotes/origin/*
> git config remote.origin.url https://gitbox.apache.org/repos/asf/bigtop.git
> # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* #
> timeout=10
> git config remote.origin.url https://gitbox.apache.org/repos/asf/bigtop.git
> # timeout=10
Fetching upstream changes from https://gitbox.apache.org/repos/asf/bigtop.git
> git fetch --tags --progress https://gitbox.apache.org/repos/asf/bigtop.git
> +refs/heads/*:refs/remotes/origin/*
Checking out Revision 51e0436e239b2bebbb4edda1b53117913b49703d
(origin/branch-1.4)
> git config core.sparsecheckout # timeout=10
> git checkout -f 51e0436e239b2bebbb4edda1b53117913b49703d
Commit message: "Bigtop 1.4.0 release"
> git rev-list --no-walk c52b9b7bc77d0b04df6530fb285c23e8fbc2db12 # timeout=10
[aarch64-slave] $ /bin/bash /tmp/jenkins5808019095228182779.sh
++ uname -m
+ ARCH=aarch64
+ case "$ARCH" in
+ POSTFIX=-aarch64
+ NETWORK='--network host'
+ cd docker/bigtop-puppet
+ sed -i 's/docker build/docker build --network host --no-cache --rm/' build.sh
+ ./build.sh 1.4.0-opensuse-42.3
++ '[' 1 '!=' 1 ']'
+++ echo 1.4.0-opensuse-42.3
+++ cut -d - -f 1
++ PREFIX=1.4.0
+++ echo 1.4.0-opensuse-42.3
+++ cut -d - -f 2
++ OS=opensuse
+++ echo 1.4.0-opensuse-42.3
+++ cut -d - -f 3
++ VERSION=42.3
+++ uname -m
++ ARCH=aarch64
++ '[' aarch64 '!=' x86_64 ']'
++ ARCH=-aarch64
++ cp ../../bigtop_toolchain/bin/puppetize.sh .
++ cat
++ docker build --network host --no-cache --rm -t
bigtop/puppet:1.4.0-opensuse-42.3-aarch64 .
Sending build context to Docker daemon 7.168kB
Step 1/4 : FROM opensuse:42.3
42.3: Pulling from library/opensuse
5f2597b0284c: Pulling fs layer
5f2597b0284c: Download complete
5f2597b0284c: Pull complete
Digest: sha256:81451bd26b48a2521db953e6376aebb5e0f55d429627017ba728219d48112f40
Status: Downloaded newer image for opensuse:42.3
---> 0fd969d14cdd
Step 2/4 : MAINTAINER dev@bigtop.apache.org
---> Running in 2a7a594f7d60
Removing intermediate container 2a7a594f7d60
---> e53a501410b5
Step 3/4 : COPY puppetize.sh /tmp/puppetize.sh
---> 20cac5eba358
Step 4/4 : RUN bash /tmp/puppetize.sh
---> Running in efda7b2cdc78
Retrieving repository 'OSS' metadata [.done]
Building repository 'OSS' cache [done]
Retrieving repository 'OSS Update' metadata [.done]
Building repository 'OSS Update' cache [done]
Loading repository data...
Reading installed packages...
'ca-certificates-mozilla' is already installed.
No update candidate for 'ca-certificates-mozilla-2.22-12.1.noarch'. The highest
available version is already installed.
'puppet' not found in package names. Trying capabilities.
'suse-release' not found in package names. Trying capabilities.
'openSUSE-release' providing 'suse-release' is already installed.
Resolving package dependencies...
Problem: systemd-sysvinit-228-27.1.aarch64 requires systemd = 228-27.1, but
this requirement cannot be provided
uninstallable providers: systemd-228-27.1.aarch64[OSS]
systemd-228-27.1.aarch64[OSS Update]
Solution 1: downgrade of systemd-228-47.1.aarch64 to systemd-228-27.1.aarch64
Solution 2: do not install systemd-sysvinit-228-27.1.aarch64
Solution 3: break systemd-sysvinit-228-27.1.aarch64 by ignoring some of its
dependencies
Choose from above solutions by number or cancel [1/2/3/c] (c): c
[91m/tmp/puppetize.sh: line 39: puppet: command not found
[0mThe command '/bin/sh -c bash /tmp/puppetize.sh' returned a non-zero code:
127
Build step 'Execute shell' marked build as failure
Finished: FAILURE
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3213) Bugs in pom.xml files when releasing Bigtop 1.4.0
Evans Ye created BIGTOP-3213: Summary: Bugs in pom.xml files when releasing Bigtop 1.4.0 Key: BIGTOP-3213 URL: https://issues.apache.org/jira/browse/BIGTOP-3213 Project: Bigtop Issue Type: Sub-task Affects Versions: 1.3.0 Reporter: Evans Ye Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3212) Deploy QFS client failed due to no such directory error
Evans Ye created BIGTOP-3212:
Summary: Deploy QFS client failed due to no such directory error
Key: BIGTOP-3212
URL: https://issues.apache.org/jira/browse/BIGTOP-3212
Project: Bigtop
Issue Type: Sub-task
Components: deployment
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
Fix For: 1.4.0
{code}
Wrapped exception:
No such file or directory - /etc/qfs/QfsClient.prp20190502-223-14p90hf[0m
[1;31mError: /Stage[main]/Qfs::Client/File[/etc/qfs/QfsClient.prp]/ensure:
change from absent to file failed: Could not set 'file' on ensure: No such file
or directory - /etc/qfs/QfsClient.prp20190502-223-14p90hf at
153:/bigtop-home/bigtop-deploy/puppet/modules/qfs/manifests/init.pp[0m
[mNotice: /Stage[main]/Qfs::Client/Package[qfs-java]/ensure: created[0m
[mNotice: /Stage[main]/Qfs::Client/File[/usr/bin/hadoop-qfs]/ensure: defined
content as '{md5}7501cd9901be7d42c803d492041db0ea'[0m
[1;31mError: Could not set 'file' on ensure: No such file or directory -
/etc/qfs/QfsClient.prp20190502-219-1w7pwqz at
153:/bigtop-home/bigtop-deploy/puppet/modules/qfs/manifests/init.pp[0m
[1;31mError: Could not set 'file' on ensure: No such file or directory -
/etc/qfs/QfsClient.prp20190502-219-1w7pwqz at
153:/bigtop-home/bigtop-deploy/puppet/modules/qfs/manifests/init.pp
{code}
Reproduce:
{code}
+ cat provisioner/docker/config_centos-7-x86_64-deploy_Bigtop-smoke-tests.yaml
docker:
memory_limit: "5g"
image: "bigtop/puppet:1.4.0-centos-7"
repo: http://repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64
distro: centos
components: [hdfs,yarn,qfs]
enable_local_repo: false
smoke_test_components: [qfs]
+ docker run --rm -v
/home/jenkins/workspace/Bigtop-trunk-smoke-tests-1.4.0/COMPONENTS/hdfs.yarn.qfs@qfs/OS/centos-7-x86_64-deploy/bigtop:/bigtop-home
bigtop/puppet:trunk-centos-7 bash -c 'chown -R 1001:1001 /bigtop-home'
+ cd provisioner/docker
+ ./docker-hadoop.sh -d
No cluster exists!
+ cd provisioner/docker
+ ./docker-hadoop.sh -C config_centos-7-x86_64-deploy_Bigtop-smoke-tests.yaml
-c 3 -s
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3210) Missing /lib/lsb/init-functions when deploying Flink on Fedora 26
Evans Ye created BIGTOP-3210:
Summary: Missing /lib/lsb/init-functions when deploying Flink on
Fedora 26
Key: BIGTOP-3210
URL: https://issues.apache.org/jira/browse/BIGTOP-3210
Project: Bigtop
Issue Type: Sub-task
Components: build
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
Fix For: 1.4.0
{code}
[mNotice: /Stage[main]/Flink::Taskmanager/Package[flink-taskmanager]/ensure:
created[0m
[1;31mError: Systemd start for flink-jobmanager failed!
journalctl log for flink-jobmanager:
-- Logs begin at Mon 2019-04-15 20:57:53 UTC, end at Mon 2019-04-15 20:58:58
UTC. --
Apr 15 20:58:55 6d782d55f4bf systemd[1]: Starting LSB: Flink jobmanager...
Apr 15 20:58:55 6d782d55f4bf flink-jobmanager[400]:
/etc/rc.d/init.d/flink-jobmanager: line 34: /lib/lsb/init-functions: No such
file or directory
Apr 15 20:58:55 6d782d55f4bf flink-jobmanager[400]:
/etc/rc.d/init.d/flink-jobmanager: line 76: log_success_msg: command not found
Apr 15 20:58:55 6d782d55f4bf flink-jobmanager[400]:
/etc/rc.d/init.d/flink-jobmanager: line 123: pidofproc: command not found
Apr 15 20:58:55 6d782d55f4bf su[420]: (to flink) root on none
Apr 15 20:58:55 6d782d55f4bf su[420]: pam_systemd(su:session): Failed to create
session: Unit systemd-logind.service is masked.
Apr 15 20:58:55 6d782d55f4bf su[420]: pam_unix(su:session): session opened for
user flink by (uid=0)
Apr 15 20:58:55 6d782d55f4bf su[420]: pam_unix(su:session): session closed for
user flink
Apr 15 20:58:58 6d782d55f4bf flink-jobmanager[400]:
/etc/rc.d/init.d/flink-jobmanager: line 123: pidofproc: command not found
Apr 15 20:58:58 6d782d55f4bf flink-jobmanager[400]:
/etc/rc.d/init.d/flink-jobmanager: line 105: log_failure_msg: command not found
Apr 15 20:58:58 6d782d55f4bf systemd[1]: flink-jobmanager.service: Control
process exited, code=exited status=127
Apr 15 20:58:58 6d782d55f4bf systemd[1]: Failed to start LSB: Flink jobmanager.
Apr 15 20:58:58 6d782d55f4bf systemd[1]: flink-jobmanager.service: Unit entered
failed state.
Apr 15 20:58:58 6d782d55f4bf systemd[1]: flink-jobmanager.service: Failed with
result 'exit-code'.
[0m
[1;31mError: /Stage[main]/Flink::Jobmanager/Service[flink-jobmanager]/ensure:
change from stopped to running failed: Systemd start for flink-jobmanager
failed!
journalctl log for flink-jobmanager:
-- Logs begin at Mon 2019-04-15 20:57:53 UTC, end at Mon 2019-04-15 20:58:58
UTC. --
Apr 15 20:58:55 6d782d55f4bf systemd[1]: Starting LSB: Flink jobmanager...
Apr 15 20:58:55 6d782d55f4bf flink-jobmanager[400]:
/etc/rc.d/init.d/flink-jobmanager: line 34: /lib/lsb/init-functions: No such
file or directory
Apr 15 20:58:55 6d782d55f4bf flink-jobmanager[400]:
/etc/rc.d/init.d/flink-jobmanager: line 76: log_success_msg: command not found
Apr 15 20:58:55 6d782d55f4bf flink-jobmanager[400]:
/etc/rc.d/init.d/flink-jobmanager: line 123: pidofproc: command not found
Apr 15 20:58:55 6d782d55f4bf su[420]: (to flink) root on none
Apr 15 20:58:55 6d782d55f4bf su[420]: pam_systemd(su:session): Failed to create
session: Unit systemd-logind.service is masked.
Apr 15 20:58:55 6d782d55f4bf su[420]: pam_unix(su:session): session opened for
user flink by (uid=0)
Apr 15 20:58:55 6d782d55f4bf su[420]: pam_unix(su:session): session closed for
user flink
Apr 15 20:58:58 6d782d55f4bf flink-jobmanager[400]:
/etc/rc.d/init.d/flink-jobmanager: line 123: pidofproc: command not found
Apr 15 20:58:58 6d782d55f4bf flink-jobmanager[400]:
/etc/rc.d/init.d/flink-jobmanager: line 105: log_failure_msg: command not found
Apr 15 20:58:58 6d782d55f4bf systemd[1]: flink-jobmanager.service: Control
process exited, code=exited status=127
Apr 15 20:58:58 6d782d55f4bf systemd[1]: Failed to start LSB: Flink jobmanager.
Apr 15 20:58:58 6d782d55f4bf systemd[1]: flink-jobmanager.service: Unit entered
failed state.
Apr 15 20:58:58 6d782d55f4bf systemd[1]: flink-jobmanager.service: Failed with
result 'exit-code'.
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3209) Revert Kafka to 0.10.2.2 and Flume to 1.8.0
Evans Ye created BIGTOP-3209: Summary: Revert Kafka to 0.10.2.2 and Flume to 1.8.0 Key: BIGTOP-3209 URL: https://issues.apache.org/jira/browse/BIGTOP-3209 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.4.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 Due to BIGTOP-3208 and the timeline of 1.4 release, I'd like to revert Kafka and Flume back to what it was in 1.3 release. A short summary to who don't know the story yet: * Kafka 2.X requires zookeeper 3.4.7+ => need to upgrade Zookeeper * Hadoop uses Zookeeper 3.4.6 => can't upgrade Zookeeper * Flume 1.9 only supports Kafka 2.X => can't upgrade Kafka to 1.X -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3207) [Provisioner] Do not exit when destroying got no cluster exists
Evans Ye created BIGTOP-3207: Summary: [Provisioner] Do not exit when destroying got no cluster exists Key: BIGTOP-3207 URL: https://issues.apache.org/jira/browse/BIGTOP-3207 Project: Bigtop Issue Type: Sub-task Components: provisioner Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 Currently the when destroy discovers no cluster exists, it exit the entire execution. However it'd be better to just exit the function and continue processing the following commands. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3203) [Sandbox ] Architecture string is a mess for Distributions
Evans Ye created BIGTOP-3203: Summary: [Sandbox ] Architecture string is a mess for Distributions Key: BIGTOP-3203 URL: https://issues.apache.org/jira/browse/BIGTOP-3203 Project: Bigtop Issue Type: Sub-task Components: sandbox Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 Tried to gather architecture string for packages in a generic way. However failed... * docker run -ti --rm bigtop/puppet:trunk-fedora-26 facter architecture => x86_64 * docker run -ti --rm bigtop/puppet:trunk-debian-9 facter architecture = > amd64 * docker run -ti --rm bigtop/slaves:trunk-debian-9-aarch64 facter architecture => aarch64 * docker run -ti --rm bigtop/slaves:trunk-fedora-26-aarch64 facter architecture => aarch64 Looks like two architecture are playing game with different rules... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3200) Bugfix Integration test framework 2.0 and documentation
Evans Ye created BIGTOP-3200: Summary: Bugfix Integration test framework 2.0 and documentation Key: BIGTOP-3200 URL: https://issues.apache.org/jira/browse/BIGTOP-3200 Project: Bigtop Issue Type: Sub-task Components: build, tests Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 There're several bugs discovered during integration. Let me fix them in this JIRA as a whole. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3199) Can not run smoke test inside docker-provisioner
Evans Ye created BIGTOP-3199: Summary: Can not run smoke test inside docker-provisioner Key: BIGTOP-3199 URL: https://issues.apache.org/jira/browse/BIGTOP-3199 Project: Bigtop Issue Type: Sub-task Components: provisioner, tests Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3198) [Sandbox] Support AARCH64/PPC64LE
Evans Ye created BIGTOP-3198: Summary: [Sandbox] Support AARCH64/PPC64LE Key: BIGTOP-3198 URL: https://issues.apache.org/jira/browse/BIGTOP-3198 Project: Bigtop Issue Type: Sub-task Components: sandbox Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3197) [Docker] Treat both exit code 0 and 2 as success for Puppet execution
Evans Ye created BIGTOP-3197: Summary: [Docker] Treat both exit code 0 and 2 as success for Puppet execution Key: BIGTOP-3197 URL: https://issues.apache.org/jira/browse/BIGTOP-3197 Project: Bigtop Issue Type: Sub-task Components: docker Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3196) Drop Apache Crunch in Bigtop distribution
Evans Ye created BIGTOP-3196: Summary: Drop Apache Crunch in Bigtop distribution Key: BIGTOP-3196 URL: https://issues.apache.org/jira/browse/BIGTOP-3196 Project: Bigtop Issue Type: Sub-task Reporter: Evans Ye Assignee: Evans Ye Triggered by Crunch compatibility issue with Spark 2. We see no much interest in keep maintaining Crunch. Plus Crunch haven't do a release for 2 years. The discussion was made in BIGTOP-3192. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3195) Add pkgs-ind to package all artifacts inside docker
Evans Ye created BIGTOP-3195: Summary: Add pkgs-ind to package all artifacts inside docker Key: BIGTOP-3195 URL: https://issues.apache.org/jira/browse/BIGTOP-3195 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3193) Add REPO and MEMORY as options for docker-provisioner
Evans Ye created BIGTOP-3193: Summary: Add REPO and MEMORY as options for docker-provisioner Key: BIGTOP-3193 URL: https://issues.apache.org/jira/browse/BIGTOP-3193 Project: Bigtop Issue Type: Sub-task Components: provisioner Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3191) Old Gradle version causes Kafka 2.1.1 build failed
Evans Ye created BIGTOP-3191:
Summary: Old Gradle version causes Kafka 2.1.1 build failed
Key: BIGTOP-3191
URL: https://issues.apache.org/jira/browse/BIGTOP-3191
Project: Bigtop
Issue Type: Sub-task
Components: build
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
Fix For: 1.4.0
{code}
* What went wrong:
A problem occurred evaluating root project 'kafka-2.1.1-src'.
> Could not find method annotationProcessor() for arguments
> [org.openjdk.jmh:jmh-generator-annprocess:1.21] on object of type
> org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output.
BUILD FAILED in 52s
error: Bad exit status from /var/tmp/rpm-tmp.gxsHho (%build)
Bad exit status from /var/tmp/rpm-tmp.gxsHho (%build)
RPM build errors:
> Task :kafka-rpm FAILED
:kafka-rpm (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 55.286
secs.
FAILURE: Build failed with an exception.
* Where:
Script '/bigtop/packages.gradle' line: 500
* What went wrong:
Execution failed for task ':kafka-rpm'.
> Process 'command 'rpmbuild'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at 11 actionable tasks: 11 executed
VCS Checkout Cache (/bigtop/.gradle/vcs-1) removing files not accessed on or
after Thu Mar 07 21:42:36 UTC 2019.
VCS Checkout Cache (/bigtop/.gradle/vcs-1) cleaned up in 0.01 secs.
https://help.gradle.org
BUILD FAILED in 1m 58s
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3190) Compatibility issue for Zeppelin with Spark 2.2.3
Evans Ye created BIGTOP-3190: Summary: Compatibility issue for Zeppelin with Spark 2.2.3 Key: BIGTOP-3190 URL: https://issues.apache.org/jira/browse/BIGTOP-3190 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3186) Bump Spark to 2.2.3
Evans Ye created BIGTOP-3186: Summary: Bump Spark to 2.2.3 Key: BIGTOP-3186 URL: https://issues.apache.org/jira/browse/BIGTOP-3186 Project: Bigtop Issue Type: Sub-task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3187) Bump Flume to 1.8.0
Evans Ye created BIGTOP-3187: Summary: Bump Flume to 1.8.0 Key: BIGTOP-3187 URL: https://issues.apache.org/jira/browse/BIGTOP-3187 Project: Bigtop Issue Type: Sub-task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3185) Bump Kafka to 2.1.1
Evans Ye created BIGTOP-3185: Summary: Bump Kafka to 2.1.1 Key: BIGTOP-3185 URL: https://issues.apache.org/jira/browse/BIGTOP-3185 Project: Bigtop Issue Type: Sub-task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 Kafka 1.1.1 won't be compatible with Flume. Let's bump it up to 2.X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3183) Zeppelin build failed
Evans Ye created BIGTOP-3183: Summary: Zeppelin build failed Key: BIGTOP-3183 URL: https://issues.apache.org/jira/browse/BIGTOP-3183 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3182) Helper message and documentation of new XXX-ind features
Evans Ye created BIGTOP-3182: Summary: Helper message and documentation of new XXX-ind features Key: BIGTOP-3182 URL: https://issues.apache.org/jira/browse/BIGTOP-3182 Project: Bigtop Issue Type: Sub-task Components: documentation Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3181) Build as non-root for COMPONENT-pkg-ind
Evans Ye created BIGTOP-3181: Summary: Build as non-root for COMPONENT-pkg-ind Key: BIGTOP-3181 URL: https://issues.apache.org/jira/browse/BIGTOP-3181 Project: Bigtop Issue Type: Sub-task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3180) Gradle 5 incompatible issue
Evans Ye created BIGTOP-3180:
Summary: Gradle 5 incompatible issue
Key: BIGTOP-3180
URL: https://issues.apache.org/jira/browse/BIGTOP-3180
Project: Bigtop
Issue Type: Sub-task
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
After upgraded to gradle 5.2 in BIGTOP-3177. There are incompatible issue for
gradle filetree API:
{code}
> Task :bigtop-tomcat-sdeb FAILED
Task ':bigtop-tomcat-sdeb' is not up-to-date because:
Task has not declared any outputs despite executing actions.
Custom actions are attached to task ':bigtop-tomcat-sdeb'.
Starting process 'command 'tar''. Working directory:
/bigtop/build/bigtop-tomcat/deb/bigtop-tomcat-6.0.45 Command: tar
--strip-components 1 -xf
/bigtop/build/bigtop-tomcat/deb/bigtop-tomcat-6.0.45/../bigtop-tomcat_6.0.45.orig.tar.gz
Successfully started process 'command 'tar''
:bigtop-tomcat-sdeb (Thread[Execution worker for ':',5,main]) completed. Took
0.135 secs.
FAILURE: Build failed with an exception.
* Where:
Script '/bigtop/packages.gradle' line: 400
* What went wrong:
Execution failed for task ':bigtop-tomcat-sdeb'.
> No signature of method:
> org.gradle.api.internal.file.collections.DefaultConfigurableFileTree.copy()
> is applicable for argument types:
> (packages_3u7wau7t5reuo24t3ysv29i35$_run_closure11$_closure27$_closure58)
> values:
> [packages_3u7wau7t5reuo24t3ysv29i35$_run_closure11$_closure27$_closure58@f15af2e]
Possible solutions: chop([I), sort(), any(), drop(int),
join(java.lang.String), tap(groovy.lang.Closure)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3179) Connection refused by www-us.apache.org mirror site
Evans Ye created BIGTOP-3179: Summary: Connection refused by www-us.apache.org mirror site Key: BIGTOP-3179 URL: https://issues.apache.org/jira/browse/BIGTOP-3179 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.1 Since yesterday we start to get connection refused by www-us.apache.org. The site seems still working but becoming very unstable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3177) [Build] Exit w/o container left for for running docker in gradle wrapper
Evans Ye created BIGTOP-3177: Summary: [Build] Exit w/o container left for for running docker in gradle wrapper Key: BIGTOP-3177 URL: https://issues.apache.org/jira/browse/BIGTOP-3177 Project: Bigtop Issue Type: Sub-task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.1 Currently Ctrl-C the COMPONENT-pkg-ind build process will have a dangling docker container left running on the build host, which leads to a bad user experience. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3176) [Build] Support non X86 archs for COMPONENT-pkg-ind
Evans Ye created BIGTOP-3176: Summary: [Build] Support non X86 archs for COMPONENT-pkg-ind Key: BIGTOP-3176 URL: https://issues.apache.org/jira/browse/BIGTOP-3176 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3175) [Build] Support to specify commit SHA1 when building packages
Evans Ye created BIGTOP-3175: Summary: [Build] Support to specify commit SHA1 when building packages Key: BIGTOP-3175 URL: https://issues.apache.org/jira/browse/BIGTOP-3175 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3174) ./gradlew repo-ind failed on Bigtop build slaves
Evans Ye created BIGTOP-3174:
Summary: ./gradlew repo-ind failed on Bigtop build slaves
Key: BIGTOP-3174
URL: https://issues.apache.org/jira/browse/BIGTOP-3174
Project: Bigtop
Issue Type: Sub-task
Components: build
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
Fix For: 1.3.1
While it was good running on Mac OS when developing BIGTOP-3009. This feature
failed to run on Bigtop build slaves(Amazon Linux). The error log:
{code}
16:56:51.685 [INFO] [org.gradle.cache.internal.DefaultCacheAccess] Creating new
cache for fileHashes, path /bigtop-home/.gradle/4.3/fileHashes/fileHashes.bin,
access org.gradle.cache.internae
16:56:51.686 [DEBUG]
[org.gradle.api.internal.changedetection.state.InMemoryCacheDecoratorFactory]
Creating in-memory store for cache
/bigtop-home/.gradle/4.3/fileHashes/fileHashes.bin (max )
16:56:51.686 [DEBUG]
[org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Acquiring file
lock for file hash cache (/bigtop-home/.gradle/4.3/fileHashes)
16:56:51.687 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting
to acquire exclusive lock on file hash cache
(/bigtop-home/.gradle/4.3/fileHashes).
16:56:51.688 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] The
file lock is held by a different Gradle process (pid: 8237, lockId:
5389536953551787496). Pinged owner at port 4175
16:56:52.770 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] The
file lock is held by a different Gradle process (pid: 8237, lockId:
5389536953551787496). Pinged owner at port 417
16:56:53.822 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] The
file lock is held by a different Gradle process (pid: 8237, lockId:
5389536953551787496). Pinged owner at port 417
{code}
I guess it is because Docker on Mac OS is running inside a hidden VM, which
isolates the lock on file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3173) Fix bugs and improve usability of COMPONENT-pkg-ind
Evans Ye created BIGTOP-3173: Summary: Fix bugs and improve usability of COMPONENT-pkg-ind Key: BIGTOP-3173 URL: https://issues.apache.org/jira/browse/BIGTOP-3173 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3172) [Provisioner] Support to specify components and smoke-tests when launching Docker Provisioner
Evans Ye created BIGTOP-3172: Summary: [Provisioner] Support to specify components and smoke-tests when launching Docker Provisioner Key: BIGTOP-3172 URL: https://issues.apache.org/jira/browse/BIGTOP-3172 Project: Bigtop Issue Type: Sub-task Components: provisioner Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3171) Update Kafka Puppet module for version 1.1.1
Evans Ye created BIGTOP-3171: Summary: Update Kafka Puppet module for version 1.1.1 Key: BIGTOP-3171 URL: https://issues.apache.org/jira/browse/BIGTOP-3171 Project: Bigtop Issue Type: Sub-task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3170) Loop and wait for daemon to be started up for Alluxio
Evans Ye created BIGTOP-3170: Summary: Loop and wait for daemon to be started up for Alluxio Key: BIGTOP-3170 URL: https://issues.apache.org/jira/browse/BIGTOP-3170 Project: Bigtop Issue Type: Sub-task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.1 It seems that the daemon startup can still fail for Alluxio. Let's try a more robust way to detect it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3169) Enable local repo is not working on yum systems
Evans Ye created BIGTOP-3169: Summary: Enable local repo is not working on yum systems Key: BIGTOP-3169 URL: https://issues.apache.org/jira/browse/BIGTOP-3169 Project: Bigtop Issue Type: Sub-task Components: deployment Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3167) Nexus is always enabled for ./gradlew XXX-ind
Evans Ye created BIGTOP-3167: Summary: Nexus is always enabled for ./gradlew XXX-ind Key: BIGTOP-3167 URL: https://issues.apache.org/jira/browse/BIGTOP-3167 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.1 The previous implementation of nexus configuration has a bug: https://github.com/apache/bigtop/blob/master/packages.gradle#L619-L626 Which leads to nexus is always enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3166) Update Kerberos module to comply with Puppet's syntax
Evans Ye created BIGTOP-3166:
Summary: Update Kerberos module to comply with Puppet's syntax
Key: BIGTOP-3166
URL: https://issues.apache.org/jira/browse/BIGTOP-3166
Project: Bigtop
Issue Type: Sub-task
Components: deployment
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
Fix For: 1.3.1
Kerberos module uses site as class name, which is a reserved keyword in puppet.
An error log thrown if deploy with Puppet 5 (BIGTOP-3161):
{code}
Error: Evaluation Error: Error while evaluating a Resource Statement, 'class'
keyword not allowed at this location (file:
/bigtop-home/bigtop-deploy/puppet/modules/kerberos/manifests/init.pp, line: 26,
column: 3) (file: /bigtop-home/bigtop-deploy/puppet/manifests/cluster.pp, line:
227) on node 02c04acdc56b.bigtop.apache.org
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3165) Bumop Spark to 2.4.0
Evans Ye created BIGTOP-3165: Summary: Bumop Spark to 2.4.0 Key: BIGTOP-3165 URL: https://issues.apache.org/jira/browse/BIGTOP-3165 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3164) Bump Kafka to 1.1.1
Evans Ye created BIGTOP-3164: Summary: Bump Kafka to 1.1.1 Key: BIGTOP-3164 URL: https://issues.apache.org/jira/browse/BIGTOP-3164 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3162) Define Bigtop 1.3.1 release BOM
Evans Ye created BIGTOP-3162: Summary: Define Bigtop 1.3.1 release BOM Key: BIGTOP-3162 URL: https://issues.apache.org/jira/browse/BIGTOP-3162 Project: Bigtop Issue Type: Task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye According to the [https://lists.apache.org/thread.html/beebf03514f1bbde1f94f6544dbfdeb49144afbb1ccd2996b81c22b5@%3Cdev.bigtop.apache.org%3E|discussion thread], the 1.3.1 release is focus on harden the overall quality of Apache Bigtop and provide a solid foundation to move forward with minimal overhead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3163) Bump Hadoop to 2.8.5
Evans Ye created BIGTOP-3163: Summary: Bump Hadoop to 2.8.5 Key: BIGTOP-3163 URL: https://issues.apache.org/jira/browse/BIGTOP-3163 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3161) Upgrade to Puppet 5.X
Evans Ye created BIGTOP-3161: Summary: Upgrade to Puppet 5.X Key: BIGTOP-3161 URL: https://issues.apache.org/jira/browse/BIGTOP-3161 Project: Bigtop Issue Type: Sub-task Components: build, deployment Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye The puppet version for Bigtop Puppet is quite old and inconsistent across platforms. Fore example, we have 3.6 on centos 7 but 4.6 on fedora, which cause various of compatibility problems. One recent example is deploying Alluxio BIGTOP-3159. It's time to bump the version of puppet up to a recent one, and sync the puppet version across platforms supported by Bigtop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3160) [Provisioner] Ignore exit code == 2 (WARNINGS) failure for Puppet execution
Evans Ye created BIGTOP-3160: Summary: [Provisioner] Ignore exit code == 2 (WARNINGS) failure for Puppet execution Key: BIGTOP-3160 URL: https://issues.apache.org/jira/browse/BIGTOP-3160 Project: Bigtop Issue Type: Sub-task Components: provisioner Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye With BIGTOP-3150, when exit code != 0, we'll raise a failure. However this also includes successful execution with warnings, which has exit code = 2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3159) Failed to deploy Alluxio due to Puppet compatibility issue
Evans Ye created BIGTOP-3159:
Summary: Failed to deploy Alluxio due to Puppet compatibility issue
Key: BIGTOP-3159
URL: https://issues.apache.org/jira/browse/BIGTOP-3159
Project: Bigtop
Issue Type: Sub-task
Components: deployment
Affects Versions: 1.3.0
Reporter: Evans Ye
The snippets to deploy Alluxio is not compatible under different Puppet
versions. On CentOS 7 it is 3.X, hence we got the error:
{code}
[mNotice: Roles to deploy: [alluxio-master, alluxio-worker, namenode,
datanode][0m
[1;31mError: Failed to parse template alluxio/alluxio-site.properties:
Filepath: /usr/share/ruby/vendor_ruby/puppet/parser/scope.rb
Line: 797
Detail: undefined method `call_function' for
#
on node de28276ab851.bigtop.apache.org[0m
{code}
The same deploy snippets works fine on Debian/Ubuntu, which has Puppet 4.X.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3158) [Test] Fix test specification with maven failsafe plugin
Evans Ye created BIGTOP-3158: Summary: [Test] Fix test specification with maven failsafe plugin Key: BIGTOP-3158 URL: https://issues.apache.org/jira/browse/BIGTOP-3158 Project: Bigtop Issue Type: Sub-task Components: tests Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Maven failsafe plugin has the way to specify which test to run: https://maven.apache.org/surefire/maven-failsafe-plugin/examples/single-test.html However, the version bigtop is currently using has bug, hence the function is not working. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3157) Failed to start Ambari server and agent on Fedora 26
Evans Ye created BIGTOP-3157: Summary: Failed to start Ambari server and agent on Fedora 26 Key: BIGTOP-3157 URL: https://issues.apache.org/jira/browse/BIGTOP-3157 Project: Bigtop Issue Type: Sub-task Components: deployment Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Ref: * https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests_BIGTOP-2996/COMPONENTS=ambari.bigtop-utils@ambari,OS=fedora-26-aarch64-deploy/5/console * https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests_BIGTOP-2996/COMPONENTS=ambari.bigtop-utils@ambari,OS=fedora-26-x86_64-deploy/5/console -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3155) Setup manual smoke test for Pull Requests
Evans Ye created BIGTOP-3155: Summary: Setup manual smoke test for Pull Requests Key: BIGTOP-3155 URL: https://issues.apache.org/jira/browse/BIGTOP-3155 Project: Bigtop Issue Type: Sub-task Components: ci Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Currently the review process for committers is quite tedious. With the scope of fully functional smoke test available now, we can compare the result of a patch before and after applying it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3154) [Provisioner] Failed to bootstrap provisioning environment on Debian/Ubuntu
Evans Ye created BIGTOP-3154:
Summary: [Provisioner] Failed to bootstrap provisioning
environment on Debian/Ubuntu
Key: BIGTOP-3154
URL: https://issues.apache.org/jira/browse/BIGTOP-3154
Project: Bigtop
Issue Type: Sub-task
Components: provisioner
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
On Debian/Ubuntu systems, the boostrap script failed to install required
packages because of outdated repo. See example:
{code}
Reading package lists...Reading package lists...Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
udev
The following NEW packages will be installed:
rng-tools udev
Building dependency tree...
Building dependency tree...
Reading state information...
Reading state information...
The following additional packages will be installed:
udev
The following NEW packages will be installed:
rng-tools udev
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 1158 kB of archives.
After this operation, 7357 kB of additional disk space will be used.
Err:1 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 udev amd64
232-25+deb9u4
404 Not Found
The following additional packages will be installed:
udev
Get:2 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 rng-tools
amd64 2-unofficial-mt.14-1+b2 [43.1 kB]
Fetched 43.1 kB in 0s (136 kB/s)
E: Failed to fetch
http://cdn-fastly.deb.debian.org/debian/pool/main/s/systemd/udev_232-25+deb9u4_amd64.deb
404 Not Found
E: Unable to fetch some archives, maybe run apt-get update or try with
--fix-missing?
sed: can't read /etc/default/rng-tools: No such file or directory
The following NEW packages will be installed:
rng-tools udev
Failed to start rng-tools.service: Unit rng-tools.service not found.
Reading package lists...0 upgraded, 2 newly installed, 0 to remove and 0 not
upgraded.
Need to get 1158 kB of archives.
After this operation, 7357 kB of additional disk space will be used.
Err:1 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 udev amd64
232-25+deb9u4
404 Not Found
Get:2 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 rng-tools
amd64 2-unofficial-mt.14-1+b2 [43.1 kB]
Fetched 43.1 kB in 0s (207 kB/s)
E: Failed to fetch
http://cdn-fastly.deb.debian.org/debian/pool/main/s/systemd/udev_232-25+deb9u4_amd64.deb
404 Not Found
E: Unable to fetch some archives, maybe run apt-get update or try with
--fix-missing?
sed: can't read /etc/default/rng-tools: No such file or directory
Failed to start rng-tools.service: Unit rng-tools.service not found.
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 1158 kB of archives.
After this operation, 7357 kB of additional disk space will be used.
Err:1 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 udev amd64
232-25+deb9u4
404 Not Found
Reading package lists...Get:2 http://cdn-fastly.deb.debian.org/debian
stretch/main amd64 rng-tools amd64 2-unofficial-mt.14-1+b2 [43.1 kB]
E: Failed to fetch
http://cdn-fastly.deb.debian.org/debian/pool/main/s/systemd/udev_232-25+deb9u4_amd64.deb
404 Not Found
E: Unable to fetch some archives, maybe run apt-get update or try with
--fix-missing?
Fetched 43.1 kB in 0s (134 kB/s)
sed: can't read /etc/default/rng-tools: No such file or directory
Failed to start rng-tools.service: Unit rng-tools.service not found.
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libcurl3-gnutls
The following NEW packages will be installed:
apt-transport-https libcurl3-gnutls
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 460 kB of archives.
After this operation, 873 kB of additional disk space will be used.
Get:1 http://cdn-fastly.deb.debian.org/debian stretch/main amd64
apt-transport-https amd64 1.4.8 [171 kB]
Building dependency tree...
Reading state information...
Building dependency tree...The following additional packages will be installed:
libcurl3-gnutls
The following NEW packages will be installed:
apt-transport-https libcurl3-gnutls
Reading state information...
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 460 kB of archives.
After this operation, 873 kB of additional disk space will be used.
Err:1 http://security-cdn.debian.org/debian-security stretch/updates/main amd64
libcurl3-gnutls amd64 7.52.1-5+deb9u7
404 Not Found
Err:2 http://security-cdn.debian.org/debian-security stretch/updates/main amd64
libcurl3-gnutls amd64 7.52.1-5+deb9u7
404 Not Found
Fetched 171 kB in 0s (310 kB/s)
E: Failed to fetch
http://security-cdn.debian.org/debian-security/pool/updates/main/c/curl/libcurl3-gnutls_7.52.1-5+deb9u7_amd64.deb
404 Not Found
E: Unable to fetch some archives, maybe run
[jira] [Created] (BIGTOP-3153) Switch to docker cp based solution to build inside containers
Evans Ye created BIGTOP-3153: Summary: Switch to docker cp based solution to build inside containers Key: BIGTOP-3153 URL: https://issues.apache.org/jira/browse/BIGTOP-3153 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye There're several drawbacks using for current bigtop-ci/build.sh script: 1. Need to build and remove image for each build 2. If we do not remove the built image and try to rely on docker build cache, the requirements to reuse the cache is very high, hence most of the time the cache is discarded. 3. A newly built image need to re-download gradle/maven stuffs, which is redundant. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3150) [Provisioner] Raise proper exit code and error messages when provisioning failed
Evans Ye created BIGTOP-3150: Summary: [Provisioner] Raise proper exit code and error messages when provisioning failed Key: BIGTOP-3150 URL: https://issues.apache.org/jira/browse/BIGTOP-3150 Project: Bigtop Issue Type: Sub-task Components: provisioner Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye When running ./gradlew docker-provisioner, the result will always be BUILD SUCCESSFUL, even if the provisioning failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3149) Support to enable Nexus proxy for Docker Provisioner
Evans Ye created BIGTOP-3149: Summary: Support to enable Nexus proxy for Docker Provisioner Key: BIGTOP-3149 URL: https://issues.apache.org/jira/browse/BIGTOP-3149 Project: Bigtop Issue Type: Sub-task Reporter: Evans Ye Assignee: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3148) Error out when configure-nexus encounter any issue
Evans Ye created BIGTOP-3148: Summary: Error out when configure-nexus encounter any issue Key: BIGTOP-3148 URL: https://issues.apache.org/jira/browse/BIGTOP-3148 Project: Bigtop Issue Type: Sub-task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Currently running ./gradlew configure-nexus always returns 0 because the underlying command is curl. For http calls getting 404 or any other non 200 returns, we shall error out so that user may know something is wrong. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3147) Nexus proxy cache configuration is broken
Evans Ye created BIGTOP-3147:
Summary: Nexus proxy cache configuration is broken
Key: BIGTOP-3147
URL: https://issues.apache.org/jira/browse/BIGTOP-3147
Project: Bigtop
Issue Type: Sub-task
Reporter: Evans Ye
To utilize nexus proxy to cache maven artifacts,
1. docker run -d -p 8081:8081 --name nexus sonatype/nexus:oss
2. ./gradlew configure-nexus
3. ./gradlew XXX-pkg # to build XXX package with nexus cache
However, configure-nexus failed silently:
{code}
curl -o /dev/null -X POST -d @/tmp/temp6599511731001671196.tmp --header
Content-Type: application/json -u admin:admin123
http://localhost:8081/nexus/service/local/repositories
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
0curl: (6) Could not resolve host: application; Unknown error
404 - Not Found
http://localhost:8081/nexus/favicon.png;>
http://localhost:8081/nexus/static/css/Sonatype-content.css?2.14.10-01;
type="text/css" media="screen" title="no title" charset="utf-8">
404 - Not Found
Repository Not Found
*strong text*
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3146) Fix run_itest.sh script
Evans Ye created BIGTOP-3146: Summary: Fix run_itest.sh script Key: BIGTOP-3146 URL: https://issues.apache.org/jira/browse/BIGTOP-3146 Project: Bigtop Issue Type: Sub-task Components: tests Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye gradlew is not available in bigtop-tests/smoke-tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3144) [Test] QFS smoke test failed on multi-node cluster
Evans Ye created BIGTOP-3144: Summary: [Test] QFS smoke test failed on multi-node cluster Key: BIGTOP-3144 URL: https://issues.apache.org/jira/browse/BIGTOP-3144 Project: Bigtop Issue Type: Sub-task Components: tests Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3143) Better way to detect whether daemon is up and running
Evans Ye created BIGTOP-3143: Summary: Better way to detect whether daemon is up and running Key: BIGTOP-3143 URL: https://issues.apache.org/jira/browse/BIGTOP-3143 Project: Bigtop Issue Type: Bug Components: build Reporter: Evans Ye Most of the daemon scripts implemented in Bigtop are using sleep to wait for process to be ready: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/alluxio/alluxio-master.svc#L42-L53 Which is not a good idea since it's not deterministic. Some discussion for a better solution is made in this PR: https://github.com/apache/bigtop/pull/442 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3142) Better user expereince when calling docker-hadoop.sh -d with no cluster running
Evans Ye created BIGTOP-3142:
Summary: Better user expereince when calling docker-hadoop.sh -d
with no cluster running
Key: BIGTOP-3142
URL: https://issues.apache.org/jira/browse/BIGTOP-3142
Project: Bigtop
Issue Type: Sub-task
Components: provisioner
Affects Versions: 1.3.0
Reporter: Evans Ye
Currently calling docker-hadoop.sh -d will just error out with a confusing
message:
{code}
$ ./docker-hadoop.sh -d
Error: No such container: bash
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3141) Release number is not honored by the build system
Evans Ye created BIGTOP-3141: Summary: Release number is not honored by the build system Key: BIGTOP-3141 URL: https://issues.apache.org/jira/browse/BIGTOP-3141 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3140) [Test] TestHadoopExamples.groovy can have $USER = null when running in Docker
Evans Ye created BIGTOP-3140: Summary: [Test] TestHadoopExamples.groovy can have $USER = null when running in Docker Key: BIGTOP-3140 URL: https://issues.apache.org/jira/browse/BIGTOP-3140 Project: Bigtop Issue Type: Sub-task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3139) Failed to deploy Ambari on CentOS/Fedora cause no service command available
Evans Ye created BIGTOP-3139:
Summary: Failed to deploy Ambari on CentOS/Fedora cause no service
command available
Key: BIGTOP-3139
URL: https://issues.apache.org/jira/browse/BIGTOP-3139
Project: Bigtop
Issue Type: Sub-task
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
Log:
{code}
[1;31mError: /Stage[main]/Ambari::Server/Exec[server setup]/returns: change
from notrun to 0 failed: Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
WARNING: Could not run /usr/sbin/sestatus: OK
Customize user account for ambari-server daemon [y/n] (n)?
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
WARNING: JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/ must be valid on
ALL hosts
WARNING: JCE Policy files are required for configuring Kerberos security. If
you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction
Policy Files are valid on all hosts.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n]
(n)?
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)?
Configuring database...
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Running initdb: This may take up to a minute.
Initializing database ... OK
About to start PostgreSQL
ERROR: Exiting with exit code 127.
REASON: Unable to start PostgreSQL server. Exiting
[0m
[mNotice: /Stage[main]/Ambari::Server/Service[ambari-server]: Dependency
Exec[server setup] has failures: true[0m
[1;31mWarning: /Stage[main]/Ambari::Server/Service[ambari-server]: Skipping
because of failed dependencies[0m
[mNotice: /Stage[main]/Ambari::Agent/Service[ambari-agent]/ensure: ensure
changed 'stopped' to 'running'[0m
[mNotice: Finished catalog run in 54.02 seconds[0m
{code}
Ref:
https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests_BIGTOP-3122/COMPONENTS=ambari.bigtop-utils@ambari,OS=centos-7-x86_64-deploy/26/console
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3138) Maven installation failed on debian-9
Evans Ye created BIGTOP-3138:
Summary: Maven installation failed on debian-9
Key: BIGTOP-3138
URL: https://issues.apache.org/jira/browse/BIGTOP-3138
Project: Bigtop
Issue Type: Sub-task
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
Ref:
https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests_BIGTOP-3122/COMPONENTS=hdfs.yarn.apex@apex,OS=debian-9-amd64-deploy/26/console
Log:
{code}
[mNotice: /Stage[main]/Bigtop_toolchain::Maven/Exec[/usr/bin/gpg -v --verify
--auto-key-retrieve --keyserver hkp://keyserver.ubuntu.com:80
apache-maven-3.5.4-bin.tar.gz.asc]/returns: gpg: cannot open '/dev/tty': No
such device or address[0m
[1;31mError: /usr/bin/gpg -v --verify --auto-key-retrieve --keyserver
hkp://keyserver.ubuntu.com:80 apache-maven-3.5.4-bin.tar.gz.asc returned 2
instead of one of [0][0m
[1;31mError: /Stage[main]/Bigtop_toolchain::Maven/Exec[/usr/bin/gpg -v
--verify --auto-key-retrieve --keyserver hkp://keyserver.ubuntu.com:80
apache-maven-3.5.4-bin.tar.gz.asc]/returns: change from notrun to 0 failed:
/usr/bin/gpg -v --verify --auto-key-retrieve --keyserver
hkp://keyserver.ubuntu.com:80 apache-maven-3.5.4-bin.tar.gz.asc returned 2
instead of one of [0][0m
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3136) [Provisioner] Local built repo should have higher priority
Evans Ye created BIGTOP-3136: Summary: [Provisioner] Local built repo should have higher priority Key: BIGTOP-3136 URL: https://issues.apache.org/jira/browse/BIGTOP-3136 Project: Bigtop Issue Type: Sub-task Components: deployment Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye In BIGTOP-2796 we pin repo installed by puppet to 900 to workaround zookeeper issue in Ubuntu. Here we set the local repo to have higher priority to obtain better testing experience. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3134) [Puppet] An upgrade to deploy Alluxio 1.8.1
Evans Ye created BIGTOP-3134: Summary: [Puppet] An upgrade to deploy Alluxio 1.8.1 Key: BIGTOP-3134 URL: https://issues.apache.org/jira/browse/BIGTOP-3134 Project: Bigtop Issue Type: Sub-task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3133) [Puppet] Add a module to deploy bigtop-utils
Evans Ye created BIGTOP-3133: Summary: [Puppet] Add a module to deploy bigtop-utils Key: BIGTOP-3133 URL: https://issues.apache.org/jira/browse/BIGTOP-3133 Project: Bigtop Issue Type: Sub-task Components: deployment Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3132) Bump Alluxio up to 1.8.1
Evans Ye created BIGTOP-3132: Summary: Bump Alluxio up to 1.8.1 Key: BIGTOP-3132 URL: https://issues.apache.org/jira/browse/BIGTOP-3132 Project: Bigtop Issue Type: Sub-task Components: build Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3131) Apex smoke test requires mvn to compile test jar
Evans Ye created BIGTOP-3131: Summary: Apex smoke test requires mvn to compile test jar Key: BIGTOP-3131 URL: https://issues.apache.org/jira/browse/BIGTOP-3131 Project: Bigtop Issue Type: Sub-task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3130) Alluxio smoke test is outdated
Evans Ye created BIGTOP-3130: Summary: Alluxio smoke test is outdated Key: BIGTOP-3130 URL: https://issues.apache.org/jira/browse/BIGTOP-3130 Project: Bigtop Issue Type: Sub-task Components: tests Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3129) Failed to run QFS smoke test
Evans Ye created BIGTOP-3129: Summary: Failed to run QFS smoke test Key: BIGTOP-3129 URL: https://issues.apache.org/jira/browse/BIGTOP-3129 Project: Bigtop Issue Type: Sub-task Components: deployment, tests Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3126) [Puppet] Failed to deploy QFS due to Permission denied error at initailization
Evans Ye created BIGTOP-3126: Summary: [Puppet] Failed to deploy QFS due to Permission denied error at initailization Key: BIGTOP-3126 URL: https://issues.apache.org/jira/browse/BIGTOP-3126 Project: Bigtop Issue Type: Sub-task Components: deployment Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3124) Remove package installation in provisioner/utils/smoke-tests.sh
Evans Ye created BIGTOP-3124:
Summary: Remove package installation in
provisioner/utils/smoke-tests.sh
Key: BIGTOP-3124
URL: https://issues.apache.org/jira/browse/BIGTOP-3124
Project: Bigtop
Issue Type: Sub-task
Components: tests
Affects Versions: 1.3.0
Reporter: Evans Ye
The package installation should be removed in smoke-tests.sh:
{code}
...
if [ -f /etc/debian_version ] ; then
apt-get -y install pig hive flume mahout sqoop
else
yum install -y pig hive flume mahout sqoop
fi
...
{code}
We should have these packages installed via puppet instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3122) CI matrix for smoke tests
Evans Ye created BIGTOP-3122: Summary: CI matrix for smoke tests Key: BIGTOP-3122 URL: https://issues.apache.org/jira/browse/BIGTOP-3122 Project: Bigtop Issue Type: Sub-task Components: tests Affects Versions: 1.3.0 Reporter: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3121) [Provisioner] Can't recreate Bigtop cluster using Docker Provisioner
Evans Ye created BIGTOP-3121:
Summary: [Provisioner] Can't recreate Bigtop cluster using Docker
Provisioner
Key: BIGTOP-3121
URL: https://issues.apache.org/jira/browse/BIGTOP-3121
Project: Bigtop
Issue Type: Sub-task
Components: provisioner
Affects Versions: 1.3.0
Reporter: Evans Ye
To reproduce the issue:
* Step 0: get an amazon linux. The one I used for test (Bigtop CI node slave02)
* Amazon Linux AMI release 2018.03
* Docker version 18.06.1-ce, build e68fc7a215d7133c34aa18e3b72b4a21fd0c6136
* docker-compose version 1.23.1, build b02f1306
* Bigtop master cobebase
* Step 1: cd bigtop/provisioner/docker
* Step 2: ./docker-hadoop -c 1
* Step 3: ./docker-hadoop -d
* Step 4: ./docker-hadoop -c 1
{code}
[jenkins@ip-172-31-8-79 docker]$ ./docker-hadoop.sh -c 1
Environment check...
Check docker:
Docker version 18.06.1-ce, build e68fc7a215d7133c34aa18e3b72b4a21fd0c6136
Check docker-compose:
docker-compose version 1.23.1, build b02f1306
Check ruby:
ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
Creating 20181228_155211_r21927_bigtop_1_8d56e66d2866 ... error
ERROR: for 20181228_155211_r21927_bigtop_1_8d56e66d2866 Cannot start service
bigtop: b'cgroups: cannot find cgroup mount destination: unknown'
ERROR: for bigtop Cannot start service bigtop: b'cgroups: cannot find cgroup
mount destination: unknown'
ERROR: Encountered errors while bringing up the project.
[LOG] Docker container(s) startup failed!
{code}
Actually the docker service is now completely broken on the host:
{code}
[jenkins@ip-172-31-8-79 ~]$ docker run -ti --rm bigtop/puppet:trunk-centos-7
bash
docker: Error response from daemon: cgroups: cannot find cgroup mount
destination: unknown.
{code}
Restart docker service seems to be no help. The only way I know to recover is
to reboot the host.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3120) [Puppet] Annoying warning message from Ambari snippet
Evans Ye created BIGTOP-3120:
Summary: [Puppet] Annoying warning message from Ambari snippet
Key: BIGTOP-3120
URL: https://issues.apache.org/jira/browse/BIGTOP-3120
Project: Bigtop
Issue Type: Sub-task
Components: deployment
Affects Versions: 1.3.0
Reporter: Evans Ye
* The message:
{code}
Warning: Unrecognized escape sequence '\/' in file
/bigtop-home/bigtop-deploy/puppet/modules/ambari/manifests/init.pp at line
38:116
Warning: Unrecognized escape sequence '\/' in file
/bigtop-home/bigtop-deploy/puppet/modules/ambari/manifests/init.pp at line
38:116
{code}
* How to reproduce?
Use Docker/Vagrant provisioner to deploy any stack of Bigtop cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3119) Failed to deploy docker provisioner on Mac occasionally
Evans Ye created BIGTOP-3119:
Summary: Failed to deploy docker provisioner on Mac occasionally
Key: BIGTOP-3119
URL: https://issues.apache.org/jira/browse/BIGTOP-3119
Project: Bigtop
Issue Type: Sub-task
Components: docker, provisioner
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
The error log:
{code}
./docker-hadoop.sh -d -c 1
WARNING: The DOCKER_IMAGE variable is not set. Defaulting to a blank string.
WARNING: The MEM_LIMIT variable is not set. Defaulting to a blank string.
WARNING: The DOCKER_IMAGE variable is not set. Defaulting to a blank string.
WARNING: The MEM_LIMIT variable is not set. Defaulting to a blank string.
Stopping 20171216224653r31038_bigtop_1 ... done
WARNING: The DOCKER_IMAGE variable is not set. Defaulting to a blank string.
WARNING: The MEM_LIMIT variable is not set. Defaulting to a blank string.
Going to remove 20171216224653r31038_bigtop_1
Removing 20171216224653r31038_bigtop_1 ... done
./config/hiera.yaml
./config/hieradata/bigtop/cluster.yaml
./config/hieradata/bigtop/ha.yaml
./config/hieradata/bigtop/noha.yaml
./config/hieradata/bigtop
./config/hieradata/site.yaml
./config/hieradata
./config/hosts
./config
.provision_id
Environment check...
Check docker:
Docker version 17.09.1-ce, build 19e2cf6
Check docker-compose:
docker-compose version 1.17.1, build 6d101fb
Check ruby:
ruby 2.0.0p648 (2015-12-16 revision 53162) [universal.x86_64-darwin16]
WARNING: The scale command is deprecated. Use the up command with the --scale
flag instead.
Creating 20171217103727r15552_bigtop_1 ...
Creating 20171217103727r15552_bigtop_1 ... done
[LOG] Bigtop Puppet configurations are shared between instances, and can be
modified under config/hieradata
../../bigtop-deploy/puppet/hieradata -> ./config/hieradata
../../bigtop-deploy/puppet/hieradata/bigtop -> ./config/hieradata/bigtop
../../bigtop-deploy/puppet/hieradata/bigtop/cluster.yaml ->
./config/hieradata/bigtop/cluster.yaml
../../bigtop-deploy/puppet/hieradata/bigtop/ha.yaml ->
./config/hieradata/bigtop/ha.yaml
../../bigtop-deploy/puppet/hieradata/bigtop/noha.yaml ->
./config/hieradata/bigtop/noha.yaml
../../bigtop-deploy/puppet/hieradata/site.yaml -> ./config/hieradata/site.yaml
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
* base: ftp.stu.edu.tw
* epel: mirror.ehost.vn
* extras: ftp.stu.edu.tw
* updates: ftp.stu.edu.tw
Resolving Dependencies
--> Running transaction check
---> Package rng-tools.x86_64 0:5-11.el7 will be installed
--> Processing Dependency: libsysfs.so.2()(64bit) for package:
rng-tools-5-11.el7.x86_64
--> Running transaction check
---> Package libsysfs.x86_64 0:2.1.0-16.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
PackageArchVersion Repository Size
Installing:
rng-tools x86_64 5-11.el7base 35 k
Installing for dependencies:
libsysfs x86_64 2.1.0-16.el7base 46 k
Transaction Summary
Install 1 Package (+1 Dependent package)
Total download size: 82 k
Installed size: 214 k
Downloading packages:
Total 206 kB/s | 82 kB 00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : libsysfs-2.1.0-16.el7.x86_64 1/2
Installing : rng-tools-5-11.el7.x86_642/2
Verifying : rng-tools-5-11.el7.x86_641/2
Verifying : libsysfs-2.1.0-16.el7.x86_64 2/2
Installed:
rng-tools.x86_64 0:5-11.el7
Dependency Installed:
libsysfs.x86_64 0:2.1.0-16.el7
Complete!
local yum = false ; NOT Enabling local yum. Packages will be pulled from
remote...
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Error: Could not find data item bigtop::hadoop_head_node in any Hiera data file
and no default supplied on node 2e4226e33ac3.bigtop.apache.org
Error: Could not find data item bigtop::hadoop_head_node in any Hiera data file
and no default supplied on node 2e4226e33ac3.bigtop.apache.org
{code}
Dig further into this we can find the shared folder hieradata is not accessible
inside container.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3118) [Puppet] Failed to deploy GPDB
Evans Ye created BIGTOP-3118: Summary: [Puppet] Failed to deploy GPDB Key: BIGTOP-3118 URL: https://issues.apache.org/jira/browse/BIGTOP-3118 Project: Bigtop Issue Type: Sub-task Components: deployment Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye There're many problems within the GPDB deployment snippets: * package: pip on centos is missing; some required package are missing as well * Puppet code: wrong class dependency when it comes to node > 1 scenario * GPDB replated: many postgres options are invalid and need to be replaced * Puppet variables: need to be updated to comply with newer version of Puppet -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3117) [Puppet] Can't run spark commands for Spark on Yarn deployment
Evans Ye created BIGTOP-3117:
Summary: [Puppet] Can't run spark commands for Spark on Yarn
deployment
Key: BIGTOP-3117
URL: https://issues.apache.org/jira/browse/BIGTOP-3117
Project: Bigtop
Issue Type: Sub-task
Components: deployment, spark
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
Deployment settings:
{code}
cat vagrantconfig.yaml
...
memory_size: 4096
number_cpus: 1
box: "puppetlabs/centos-7.2-64-nocm"
repo: "http://repos.bigtop.apache.org/releases/1.3.0/centos/7/x86_64;
num_instances: 1
distro: centos
components: [hdfs, yarn, mapreduce, spark]
enable_local_repo: true
run_smoke_tests: false
smoke_test_components: [hdfs, yarn, mapreduce]
{code}
After deployed, execute spark commands:
{code}
[root@bigtop1 bin]# ./spark-shell
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/fs/FSDataInputStream
at
org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at
org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:121)
at
org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at
org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.fs.FSDataInputStream
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3116) [Provisioner] Docker Provisioner failed to deploy on mac due to missing site.yaml file
Evans Ye created BIGTOP-3116:
Summary: [Provisioner] Docker Provisioner failed to deploy on mac
due to missing site.yaml file
Key: BIGTOP-3116
URL: https://issues.apache.org/jira/browse/BIGTOP-3116
Project: Bigtop
Issue Type: Sub-task
Components: provisioner
Affects Versions: 1.3.0
Environment: error log:
{code}
Error: Evaluation Error: Error while evaluating a Function Call, Could not find
data item bigtop::hadoop_head_node in any Hiera data file and no default
supplied at /bigtop-home/bigtop-deploy/puppet/manifests/cluster.pp:171:23 on
node b1e61613af66.bigtop.apache.org
{code}
Look further inside the container:
{code}
root@b1e61613af66:~# ls -al /etc/puppet/
ls: cannot access '/etc/puppet/hiera.yaml': No such file or directory
ls: cannot access '/etc/puppet/hieradata': No such file or directory
{code]
Reporter: Evans Ye
Fix For: 1.4.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3115) [Puppet] Alluxio master/worker can not startup
Evans Ye created BIGTOP-3115:
Summary: [Puppet] Alluxio master/worker can not startup
Key: BIGTOP-3115
URL: https://issues.apache.org/jira/browse/BIGTOP-3115
Project: Bigtop
Issue Type: Sub-task
Components: deployment
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
Fix For: 1.4.0
Use Vagrant Provisioner to deploy alluxio, got the following output:
{code}
==> bigtop1: Notice: Roles to deploy: [alluxio-master, alluxio-worker]
==> bigtop1: Notice: alluxio ---> master host
==> bigtop1: Warning: Unrecognized escape sequence '\/' in file
/tmp/vagrant-puppet/modules-197812bd65c5ed589cb0fd7a86c2a130/ambari/manifests/init.pp
at line 38:116
==> bigtop1: Warning: Unrecognized escape sequence '\/' in file
/tmp/vagrant-puppet/modules-197812bd65c5ed589cb0fd7a86c2a130/ambari/manifests/init.pp
at line 38:116
==> bigtop1: Notice: Compiled catalog for bigtop1.vagrant in environment
production in 1.58 seconds
==> bigtop1: Notice: /Stage[main]/Hadoop_cluster_node/User[testuser]/ensure:
created
==> bigtop1: Notice: /Stage[main]/Hadoop_cluster_node/User[jenkins]/ensure:
created
==> bigtop1: Notice: /Stage[main]/Bigtop_repo/Yumrepo[Bigtop_0]/ensure: created
==> bigtop1: Notice: /Stage[main]/Hadoop_cluster_node/User[hudson]/ensure:
created
==> bigtop1: Notice: /Stage[main]/Jdk/Package[jdk]/ensure: created
==> bigtop1: Notice: /Stage[main]/Alluxio::Common/Package[alluxio]/ensure:
created
==> bigtop1: Notice:
/Stage[main]/Alluxio::Common/File[/etc/alluxio/conf/log4j.properties]/content:
content changed '{md5}0c6aa5b2cf0daedb77977ecc6379cffb' to
'{md5}3e1aa7be70f8cc22a75957cfdb1a90e5'
==> bigtop1: Notice:
/Stage[main]/Alluxio::Common/File[/etc/alluxio/conf/alluxio-env.sh]/content:
content changed '{md5}0bff533748d5d4f8de0517234b3e83c4' to
'{md5}3b16f1232afe746f89875b9e655e380b'
==> bigtop1: Notice: /Stage[main]/Alluxio::Master/Exec[alluxio
formatting]/returns: executed successfully
==> bigtop1: Notice: /Stage[main]/Alluxio::Worker/Service[alluxio-worker]:
Dependency Service[alluxio-master] has failures: true
==> bigtop1: Error: Could not start Service[alluxio-master]: Execution of
'/bin/systemctl start alluxio-master' returned 5: Failed to start
alluxio-master.service: Unit not found.
==> bigtop1: Wrapped exception:
==> bigtop1: Execution of '/bin/systemctl start alluxio-master' returned 5:
Failed to start alluxio-master.service: Unit not found.
==> bigtop1: Error:
/Stage[main]/Alluxio::Master/Service[alluxio-master]/ensure: change from
stopped to running failed: Could not start Service[alluxio-master]: Execution
of '/bin/systemctl start alluxio-master' returned 5: Failed to start
alluxio-master.service: Unit not found.
==> bigtop1: Warning: /Stage[main]/Alluxio::Worker/Service[alluxio-worker]:
Skipping because of failed dependencies
==> bigtop1: Notice: Finished catalog run in 69.99 seconds
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3114) [TEST] HBase Importtsv smoke tests are failing
Evans Ye created BIGTOP-3114:
Summary: [TEST] HBase Importtsv smoke tests are failing
Key: BIGTOP-3114
URL: https://issues.apache.org/jira/browse/BIGTOP-3114
Project: Bigtop
Issue Type: Sub-task
Components: tests
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
3 tests are failing:
* TestImportTsv. testBulkImportNonTsv
* TestImportTsv. testBulkImportTsv
* TestImportTsv. testLoadToNewTable
Dig a bit into it and found the test is using wrong jar file to fire up tests:
{code}
HADOOP_CLASSPATH=`hbase classpath` hadoop jar
/usr/lib/hbase/hbase-annotations-1.3.2.jar importtsv
-Dimporttsv.bulk.output=/user/root/import_movies_output
-Dimporttsv.columns=HBASE_ROW_KEY,general:title,release:date,release:video_date,IMDb:URL,genre:unknown,genre:Action,genre:Adventure,genre:Animation,genre:Children,genre:Comedy,genre:Crime,genre:Documentary,genre:Drama,genre:Fantasy,genre:Film-Noir,genre:Horror,genre:Musical,genre:Mystery,genre:Romance,genre:Sci-Fi,genre:Thriller,genre:War,genre:Western
movies /user/root/movies_tsv
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BIGTOP-3113) Update previous releases links to the archive.apache.org in the download page
Evans Ye created BIGTOP-3113: Summary: Update previous releases links to the archive.apache.org in the download page Key: BIGTOP-3113 URL: https://issues.apache.org/jira/browse/BIGTOP-3113 Project: Bigtop Issue Type: Bug Components: website Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 After old releases removed at [https://dist.apache.org/repos/dist/release/bigtop], all the download links of previous releases are broken. This Jira is to update the download links from mirrors to archive server. See BIGTOP-1162 for more context. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3112) Switch
Evans Ye created BIGTOP-3112: Summary: Switch Key: BIGTOP-3112 URL: https://issues.apache.org/jira/browse/BIGTOP-3112 Project: Bigtop Issue Type: Bug Reporter: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3111) [Provisioner] docker provisioner does not honor memory_limit setting
Evans Ye created BIGTOP-3111: Summary: [Provisioner] docker provisioner does not honor memory_limit setting Key: BIGTOP-3111 URL: https://issues.apache.org/jira/browse/BIGTOP-3111 Project: Bigtop Issue Type: Sub-task Components: provisioner Affects Versions: 1.3.0 Environment: Docker on Mac, Amazon Linux Reporter: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3110) [Puppet] Hive metastore can not startup successfully
Evans Ye created BIGTOP-3110:
Summary: [Puppet] Hive metastore can not startup successfully
Key: BIGTOP-3110
URL: https://issues.apache.org/jira/browse/BIGTOP-3110
Project: Bigtop
Issue Type: Sub-task
Components: deployment, hive
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
Using either vagrant or docker provisioner with hive component configured to
deploy, though the deployment seems successful, hive commands can not be
executed:
{code}
hive> show tables;
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
{code}
Looking further into the log /var/log/hive/hive-metastore.log
{code}
2018-12-13T07:00:17,388 WARN [main] metastore.MetaStoreDirectSql: Self-test
query [select "DB_ID" from "DBS"] failed; direct SQL is disabled
javax.jdo.JDODataStoreException: Error executing SQL query "select "DB_ID" from
"DBS"".
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
~[datanucleus-api-jdo-4.2.4.jar:?]
at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
~[datanucleus-api-jdo-4.2.4.jar:?]
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:216)
~[datanucleus-api-jdo-4.2.4.jar:?]
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.runTestQuery(MetaStoreDirectSql.java:224)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.(MetaStoreDirectSql.java:144)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:408)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:340)
[hive-exec-2.3.3.jar:2.3.3]
at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:301)
[hive-exec-2.3.3.jar:2.3.3]
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
[hadoop-common-2.8.4.jar:?]
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
[hadoop-common-2.8.4.jar:?]
at
org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:58)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:624)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:590)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:584)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:651)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:427)
[hive-exec-2.3.3.jar:2.3.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_191]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_191]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:79)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:92)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6887)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6882)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:7140)
[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:7067)
[hive-exec-2.3.3.jar:2.3.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_191]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_191]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
[hadoop-common-2.8.4.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
[hadoop-common-2.8.4.jar:?]
Caused
[jira] [Created] (BIGTOP-3109) Overhaul the deployment and testing modules
Evans Ye created BIGTOP-3109: Summary: Overhaul the deployment and testing modules Key: BIGTOP-3109 URL: https://issues.apache.org/jira/browse/BIGTOP-3109 Project: Bigtop Issue Type: Task Components: deployment, tests Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 >From release to release, packages included in Bigtop are involved as well and >most of them are diverged from what they were. This is an umbrella jira to: * Check existing codebase/features * Report and fix bugs * Build up CI examples * Document the features The scope of this task is focused on deployment and testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3108) Update version strings to 1.3.0 in the master
Evans Ye created BIGTOP-3108: Summary: Update version strings to 1.3.0 in the master Key: BIGTOP-3108 URL: https://issues.apache.org/jira/browse/BIGTOP-3108 Project: Bigtop Issue Type: Task Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 There're things updated in branch-1.3 not being brought back to master. * Branch-1.3: Update repo URLs to 1.3.0 (I didn't see the Jira thoungh) [https://github.com/apache/bigtop/commit/22f54f0e3901cc0f8228f409db4e2402060ea16d] * master: BIGTOP-3089 Let's sync them up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3105) Sync up site changes from branch 1.3 to master
Evans Ye created BIGTOP-3105: Summary: Sync up site changes from branch 1.3 to master Key: BIGTOP-3105 URL: https://issues.apache.org/jira/browse/BIGTOP-3105 Project: Bigtop Issue Type: Task Components: website Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.4.0 There're several changes left in branch-1.3. Let's sync them up in the master branch for the consistency of up coming changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3052) Maven version is fixed in security verification code
Evans Ye created BIGTOP-3052: Summary: Maven version is fixed in security verification code Key: BIGTOP-3052 URL: https://issues.apache.org/jira/browse/BIGTOP-3052 Project: Bigtop Issue Type: Bug Components: toolchain Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye In BIGTOP-2379, the maven.pp has been refactored using $mvnversion for versioning: [https://github.com/apache/bigtop/commit/d401eda870a2abfed2b897ed488ee2a571678e68#diff-f5c19b3a16072bc1633fd8db96c6698cR18] In BIGTOP-3037, secure download for maven was introduced. However, the version is fixed to 3.5.3 [https://github.com/apache/bigtop/commit/980e1883a66256b904cdcd11419f5745880ec55a#diff-f5c19b3a16072bc1633fd8db96c6698cR30] When maven version automatically upgraded (BIGTOP-3031), the version of downloaded maven and the signature diverse. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3050) cgroups: cannot found cgroup mount destination: unknown
Evans Ye created BIGTOP-3050: Summary: cgroups: cannot found cgroup mount destination: unknown Key: BIGTOP-3050 URL: https://issues.apache.org/jira/browse/BIGTOP-3050 Project: Bigtop Issue Type: Bug Components: ci, docker Affects Versions: 1.3.0 Reporter: Evans Ye Didn't look deep into this yet. But it seems that we hit docker's bug. https://github.com/docker/for-linux/issues/219 cgroups: cannot found cgroup mount destination: unknown -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3048) Revert BIGTOP-3001 to bring packaging CI back
Evans Ye created BIGTOP-3048: Summary: Revert BIGTOP-3001 to bring packaging CI back Key: BIGTOP-3048 URL: https://issues.apache.org/jira/browse/BIGTOP-3048 Project: Bigtop Issue Type: Bug Components: ci Affects Versions: 1.3.0 Reporter: Evans Ye Assignee: Evans Ye According to the discussion thread [https://s.apache.org/3KEV]. We should revert BIGTOP-3001 and bring packaging CI back. After we have properly handled the UID problem. BIGTOP-3001 can be added back to the code base. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3039) git browse and git clone are pointing to the same URL
Evans Ye created BIGTOP-3039: Summary: git browse and git clone are pointing to the same URL Key: BIGTOP-3039 URL: https://issues.apache.org/jira/browse/BIGTOP-3039 Project: Bigtop Issue Type: Bug Components: website Affects Versions: 1.2.1 Reporter: Evans Ye Assignee: Evans Ye Fix For: 1.3.0 Both browse and clone pointing to ASF git site but I don't think people now browsing git repo on it. Mostly using github instead. Meanwhile, how to clone ASF repo is not listed there. So it need to be changed as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3009) Add gradle repo-ind
Evans Ye created BIGTOP-3009: Summary: Add gradle repo-ind Key: BIGTOP-3009 URL: https://issues.apache.org/jira/browse/BIGTOP-3009 Project: Bigtop Issue Type: Sub-task Reporter: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-3008) Support configure-nexus for gradle COMPONENT-pkg-ind
Evans Ye created BIGTOP-3008: Summary: Support configure-nexus for gradle COMPONENT-pkg-ind Key: BIGTOP-3008 URL: https://issues.apache.org/jira/browse/BIGTOP-3008 Project: Bigtop Issue Type: Sub-task Reporter: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-2998) Extend Provisioner CI matrix to run smoke tests
Evans Ye created BIGTOP-2998: Summary: Extend Provisioner CI matrix to run smoke tests Key: BIGTOP-2998 URL: https://issues.apache.org/jira/browse/BIGTOP-2998 Project: Bigtop Issue Type: Sub-task Components: deployment, provisioner, tests Affects Versions: 1.2.1 Environment: https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-trunk-deployments/ Reporter: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-2997) Add AARCH64 in Provisioner CI matrix
Evans Ye created BIGTOP-2997: Summary: Add AARCH64 in Provisioner CI matrix Key: BIGTOP-2997 URL: https://issues.apache.org/jira/browse/BIGTOP-2997 Project: Bigtop Issue Type: Sub-task Reporter: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BIGTOP-2996) Update Provisioner CI matrix against Bigtop master Distros
Evans Ye created BIGTOP-2996: Summary: Update Provisioner CI matrix against Bigtop master Distros Key: BIGTOP-2996 URL: https://issues.apache.org/jira/browse/BIGTOP-2996 Project: Bigtop Issue Type: Sub-task Reporter: Evans Ye -- This message was sent by Atlassian JIRA (v7.6.3#76005)
