[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. IMPALA-8680: Docker-based tests fail to archive the minicluster component logs Inside docker container copy logs of cluster components hdfs, yarn, kudu from folder testdata/cluster/cdh/node-/var/log/ to folder logs/cluster/ Testing: - running docker-based tests and checked that minicluster logs are preserved and archived - test if minicluster logs get copied also in case when something gets wrong during build Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Reviewed-on: http://gerrit.cloudera.org:8080/15898 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M docker/entrypoint.sh 1 file changed, 66 insertions(+), 21 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Tue, 31 Aug 2021 06:58:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7440/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Tue, 31 Aug 2021 00:51:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Tue, 31 Aug 2021 00:51:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 7: Code-Review+2 Thanks for working on this -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Tue, 31 Aug 2021 00:50:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9385/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Fri, 27 Aug 2021 12:48:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Zoltan Garaguly has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 7: (1 comment) Thank you for your comments. I fixed the script to save minicluster logs even in case something gets wrong during the build, dataload or any other process in build_impdev phase. http://gerrit.cloudera.org:8080/#/c/15898/6/docker/entrypoint.sh File docker/entrypoint.sh: http://gerrit.cloudera.org:8080/#/c/15898/6/docker/entrypoint.sh@304 PS6, Line 304: # and can confuse > One thing I noticed when I ran this is that this code doesn't execute if an Done -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Fri, 27 Aug 2021 12:34:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Hello Laszlo Gaal, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15898 to look at the new patch set (#7). Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. IMPALA-8680: Docker-based tests fail to archive the minicluster component logs Inside docker container copy logs of cluster components hdfs, yarn, kudu from folder testdata/cluster/cdh/node-/var/log/ to folder logs/cluster/ Testing: - running docker-based tests and checked that minicluster logs are preserved and archived - test if minicluster logs get copied also in case when something gets wrong during build Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e --- M docker/entrypoint.sh 1 file changed, 66 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/15898/7 -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/15898/6/docker/entrypoint.sh File docker/entrypoint.sh: http://gerrit.cloudera.org:8080/#/c/15898/6/docker/entrypoint.sh@304 PS6, Line 304: copy_cluster_logs One thing I noticed when I ran this is that this code doesn't execute if any of the commands above fail. That must be because we use "set -e" when executing these functions. One way to fix that would be to follow the structure that test_suite() uses. In other words: ret = 0 if ! ./buildall.sh -noclean -format -testdata -notests; then ret = 1 echo "Build + dataload failed!" else echo "Build + dataload succeeded!" fi Then, below that point, use ret to decide what needs to do. If ret is 0, then do all the current steps. If ret is 1, skip building parquet-reader/impala-profile-tool and the hardlinking duplicate HDFS blocks, but otherwise do basically the same things. It looks like test_suite() uses a lot of "cmd || true" structure to avoid exiting, then it returns ret at the end. -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Thu, 22 Jul 2021 03:34:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Zoltan Garaguly has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 6: I rebased and re-tested the fix. Could you please review it? -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Tue, 13 Jul 2021 09:00:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 5: I'm running the docker based tests with this to look at the output, then I'll review. -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Tue, 09 Mar 2021 00:20:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8144/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Wed, 17 Feb 2021 12:52:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Hello Laszlo Gaal, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15898 to look at the new patch set (#5). Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. IMPALA-8680: Docker-based tests fail to archive the minicluster component logs Inside docker container copy logs of cluster components hdfs, yarn, kudu from folder testdata/cluster/cdh/node-/var/log/ to folder logs/cluster/ Testing: - running docker-based tests and checked that minicluster logs are preserved and archived Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e --- M docker/entrypoint.sh 1 file changed, 29 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/15898/5 -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Wed, 20 May 2020 14:50:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5864/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Wed, 20 May 2020 09:33:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6112/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Wed, 20 May 2020 09:19:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Zoltan Garaguly has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/15898/3/docker/entrypoint.sh File docker/entrypoint.sh: http://gerrit.cloudera.org:8080/#/c/15898/3/docker/entrypoint.sh@319 PS3, Line 319: find /logs -xtype > There is also another problem here (not covered in the original ticket): Fo I agree with the necessity to add copy_cluster_logs call to test_suite() function as well in order to have the cluster logs for test containers too. On the other hand I do not understand the point of creating symlinks in test containers because the goal of copy_cluster_logs function is exactly to get rid of this symlink and copy the original content (the cluster logs) in the folder where the symlink was before. The copy happens also in case when symlink was not there. -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Wed, 20 May 2020 08:27:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Hello Laszlo Gaal, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15898 to look at the new patch set (#4). Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. IMPALA-8680: Docker-based tests fail to archive the minicluster component logs Copy logs of components hdfs, yarn, kudu from folder testdata/cluster/cdh/node-/var/log/ to folder logs/cluster/ Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e --- M docker/entrypoint.sh 1 file changed, 29 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/15898/4 -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 3: (2 comments) Went through the logic; found a preexisting failure case not even described in the original ticket. http://gerrit.cloudera.org:8080/#/c/15898/3/docker/entrypoint.sh File docker/entrypoint.sh: http://gerrit.cloudera.org:8080/#/c/15898/3/docker/entrypoint.sh@319 PS3, Line 319: copy_cluster_logs I'd recommend moving this line before L317: according to its comments, that line attempts to remove the very symlinks that copy_cluster_logs" is trying to save for later (although the exact flag and predicate combination seems to be wrong there). http://gerrit.cloudera.org:8080/#/c/15898/3/docker/entrypoint.sh@319 PS3, Line 319: copy_cluster_logs There is also another problem here (not covered in the original ticket): For each instantiation of the container (whether for build or test purposes) the driver script test_with_docker.py mounts a different log directory at the containers "/log" mount point (see L580-581: https://github.com/apache/impala/blob/master/docker/test-with-docker.py#L580, L655, L815 https://github.com/apache/impala/blob/master/docker/test-with-docker.py#L815). This means anything created there from within the container will not be present in the log tree of a different container run, unless there is a mechanism in test-with-docker.py that replicates or recreates the artifact. Add to this that the cluster log symlinks are created by testdata/cluster/admin in the function create_cluster(), which is run only in the build container, but not in any of the test containers. This has the end result that none of the test containers will actually contain the symlinks from /logs/cluster to the node logs, because the test containers run only testdata/cluster/admin start_cluster(), which does not ensure that the symlinks are present. I see two ways out of this. a) either fix testdata/cluster/admin so that start_cluster() (re)creates these symlink if they are gone b) or fix entrypoint.sh to the same effect. Once this is fixed, then the call to copy_cluster_logs should also be invoked from boot_container() (or possibly from test_suite() ) to ensure that the symlinks are there to capture the log files, so that clpy_cluster_logs can later pick them up and archive them. -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Mon, 18 May 2020 17:40:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Wed, 13 May 2020 21:24:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5821/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Wed, 13 May 2020 16:31:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5817/ -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Wed, 13 May 2020 13:37:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6049/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Wed, 13 May 2020 08:38:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5817/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Wed, 13 May 2020 08:18:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Zoltan Garaguly has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/15898/1/docker/entrypoint.sh File docker/entrypoint.sh: http://gerrit.cloudera.org:8080/#/c/15898/1/docker/entrypoint.sh@356 PS1, Line 356: CDH_VERSION=`echo $x | sed "s#testdata/cluster/\(.*\)/node-.*#\1#"` > The readability of the sed scripts could be increased by changing the patte Done -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Wed, 13 May 2020 08:16:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Zoltan Garaguly has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15898 Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. IMPALA-8680: Docker-based tests fail to archive the minicluster component logs Copy logs of components hdfs, yarn, kudu from folder testdata/cluster/cdh/node-/var/log/ to folder logs/cluster/ Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e --- M docker/entrypoint.sh 1 file changed, 26 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/15898/2 -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Laszlo Gaal
