Repository: zeppelin Updated Branches: refs/heads/master 6e97a527e -> 09dc9fcae
[MINOR] Let jenkins print some build summary on failure ### What is this PR for? Currently actual CI build occurs in individual contributor's travis-ci account, and apache Jenkins poll the result from travis-ci and print build status on it's console output, like ``` Build https://travis-ci.org/test/zeppelin/builds/451036679 [1] OK https://travis-ci.org/test/zeppelin/jobs/451036680 [2] Running ... https://travis-ci.org/test/zeppelin/jobs/451036681 [3] Running ... https://travis-ci.org/test/zeppelin/jobs/451036682 [4] Not completed https://travis-ci.org/test/zeppelin/jobs/451036683 [5] Running ... https://travis-ci.org/test/zeppelin/jobs/451036684 [6] Running ... https://travis-ci.org/test/zeppelin/jobs/451036685 [7] Running ... https://travis-ci.org/test/zeppelin/jobs/451036686 [8] Not started https://travis-ci.org/test/zeppelin/jobs/451036687 [9] Not started https://travis-ci.org/test/zeppelin/jobs/451036688 [10] Not started https://travis-ci.org/test/zeppelin/jobs/451036689 [11] Not started https://travis-ci.org/test/zeppelin/jobs/451036690 1 job(s) failed, 9 job(s) running/pending ``` When there're failed job, developer has to open the travis-ci link and download the log and manually look through to see what was wrong. This PR make Jenkins not only print build status but also short summary of failure, so developer quickly identify what went wrong. After this patch, Jenkins console output will look something like ``` [2018-11-18 09:11:20] Author: test, commit: ee6d89b [2018-11-18 09:11:20] -------------------------------- [2018-11-18 09:11:25] Get build status ... Build https://travis-ci.org/test/zeppelin/builds/451036679 [1] OK https://travis-ci.org/test/zeppelin/jobs/451036680 [2] OK https://travis-ci.org/test/zeppelin/jobs/451036681 [3] Error 1 https://travis-ci.org/test/zeppelin/jobs/451036682 Tests run: 38, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 153.857 sec <<< FAILURE! - in org.apache.zeppelin.notebook.NotebookTest Please check full log at https://api.travis-ci.org/v3/job/451036682/log.txt [4] Not completed https://travis-ci.org/test/zeppelin/jobs/451036683 Your build has been stopped. Please check full log at https://api.travis-ci.org/v3/job/451036683/log.txt [5] OK https://travis-ci.org/test/zeppelin/jobs/451036684 [6] Error 1 https://travis-ci.org/test/zeppelin/jobs/451036685 Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 14.374 sec <<< FAILURE! - in org.apache.zeppelin.spark.SparkRInterpreterTest Please check full log at https://api.travis-ci.org/v3/job/451036685/log.txt [7] Error 1 https://travis-ci.org/test/zeppelin/jobs/451036686 Tests run: 15, Failures: 15, Errors: 0, Skipped: 0, Time elapsed: 216.32 sec <<< FAILURE! - in org.apache.zeppelin.interpreter.SparkIntegrationTest Please check full log at https://api.travis-ci.org/v3/job/451036686/log.txt [8] Error 1 https://travis-ci.org/test/zeppelin/jobs/451036687 Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.021 sec <<< FAILURE! - in org.apache.zeppelin.spark.SparkRInterpreterTest Please check full log at https://api.travis-ci.org/v3/job/451036687/log.txt [9] Error 1 https://travis-ci.org/test/zeppelin/jobs/451036688 Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 18.169 sec <<< FAILURE! - in org.apache.zeppelin.spark.SparkRInterpreterTest Please check full log at https://api.travis-ci.org/v3/job/451036688/log.txt [10] OK https://travis-ci.org/test/zeppelin/jobs/451036689 [11] Error 1 https://travis-ci.org/test/zeppelin/jobs/451036690 Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 353.585 sec <<< FAILURE! - in org.apache.zeppelin.livy.LivyInterpreterIT Please check full log at https://api.travis-ci.org/v3/job/451036690/log.txt 7 job(s) failed, 0 job(s) running/pending ``` ### What type of PR is it? Improvement ### How should this be tested? * try script manually against some build e.g. ``` python ./travis_check.py [github username] [commit hash] python ./travis_check.py NicolasRouquette ee6d89b ``` Author: Lee moon soo <leemoon...@gmail.com> Closes #3231 from Leemoonsoo/ci_summary and squashes the following commits: 9f42b3226 [Lee moon soo] print build log summary Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/09dc9fca Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/09dc9fca Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/09dc9fca Branch: refs/heads/master Commit: 09dc9fcae542815728758b09a32b19456134dd89 Parents: 6e97a52 Author: Lee moon soo <leemoon...@gmail.com> Authored: Sun Nov 18 09:12:30 2018 +0900 Committer: Lee moon soo <m...@apache.org> Committed: Tue Nov 20 03:53:38 2018 +0900 ---------------------------------------------------------------------- travis_check.py | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/09dc9fca/travis_check.py ---------------------------------------------------------------------- diff --git a/travis_check.py b/travis_check.py index ea5e37b..009ba9a 100644 --- a/travis_check.py +++ b/travis_check.py @@ -30,7 +30,10 @@ # # with custom check interval # python travis_check.py Leemoonsoo 1f2549a 5,60,60 -import os, sys, getopt, traceback, json, requests, time +import os, sys, getopt, traceback, json, requests, time, urllib3, re + +# disable SNIMissingWarning. see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings +urllib3.disable_warnings() author = sys.argv[1] commit = sys.argv[2] @@ -70,6 +73,51 @@ def getBuildStatus(author, commit): def status(index, msg, jobId): return '{:20}'.format("[" + str(index+1) + "] " + msg) + "https://travis-ci.org/" + author + "/zeppelin/jobs/" + str(jobId) + +# load full build log and summarize +def logSummary(url): + # test start pattern "Running org.apache.zeppelin.scheduler.ParallelSchedulerTest" + testStartPattern = re.compile("^Running[ ](.*)") + # test end pattern "Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.554 sec - in org.apache.zeppelin.scheduler.JobTest" + testEndPattern = re.compile("^Tests [^0-9]*([0-9]+)[^0-9]*([0-9]+)[^0-9]*([0-9]+)[^0-9]*([0-9]+)[^-]*[-][ ]in[ ](.*)") + + tests = {} + resp = requests.get(url=url) + lines = resp.text.splitlines() + lastNonEmptyLine = "" + indent = '{:10}'.format("") + + for line in lines: + if not len(line.strip()) == 0: + lastNonEmptyLine = line + + mStart = testStartPattern.match(line) + if mStart: + testName = mStart.group(1) + tests[testName] = { + "start": mStart + } + continue + + mEnd = testEndPattern.match(line) + if mEnd: + testName = mEnd.group(5) + tests[testName]["end"] = mEnd + continue + + for testName, test in tests.items(): + if not "end" in test: + print(indent + "Test " + testName + " never finished") + else: + failures = int(test["end"].group(2)) + errors = int(test["end"].group(3)) + if failures > 0 or errors > 0: + print(indent + test["end"].group(0)) + + if not lastNonEmptyLine.startswith("Done"): + print(indent + lastNonEmptyLine) + print(indent + "Please check full log at " + url) + def printBuildStatus(build): failure = 0 running = 0 @@ -88,11 +136,13 @@ def printBuildStatus(build): if result == None: print(status(index, "Not completed", jobId)) failure = failure + 1 + logSummary("https://api.travis-ci.org/v3/job/" + str(jobId) + "/log.txt") elif result == 0: print(status(index, "OK", jobId)) else: print(status(index, "Error " + str(result), jobId)) failure = failure + 1 + logSummary("https://api.travis-ci.org/v3/job/" + str(jobId) + "/log.txt") else: print(status(index, "Unknown state", jobId)) failure = failure + 1