Hey gang.
I’ve been thinking a lot over the past few months about the various
Jenkins jobs that we have that run after a commit. Their job is to tell us
when a commit break a unit test. They eat up an incredible amount of time on
the build infrastructure and I’m not really sure if they provide much value in
their current form. I suspect most of us pretty much ignore those emails,
however, for a variety of reasons:
a) There are simply too many of them.
b) The content of the email is hard to quickly parse/read to get useful
information.
c) The unit tests are almost always broken.
Worse yet: these jobs don’t actually test everything (all of
hadoop-tools is missing!) and skip out on some vital information.
With Apache Yetus 0.3.0, we added a mode called ‘qbt’ aka Quality Build
Tool. It effectively run Apache Yetus over the entire source tree and
generates the columnar output similar to what you would see during precommit.
I’ve been running this mode against Apache Hadoop on Jenkins for a few months
now, working out the kinks, etc, and it was working well except for one thing…
the generated report was still not great over email:
1) Couldn’t use HTML since the ASF mailing lists reject HTML-formatted mail
2) Mail clients that used proportional fonts were still a big mess
For 0.4.0, I’m working on a feature that adds a ‘brief’ mode [
YETUS-390 ] with the (long term?) intention of replacing all of these emails
that get generated by Jenkins. I’d like to hear some feedback on if a) I’m
just spinning my wheels and no one is actually interested in fixing this
problem and b) the replacement email generated either needs more info or less
info.
Here’s the currently generated email. Note that some subsystems have
been told not to fail the build due to warnings (e.g., cc, javac, javadoc,
shellcheck, whitespace, etc) but still generate a log of problems so that they
may be investigated and used for cleanup.
Thanks!
==== cut here =====
For more details, see https://builds.apache.org/job/hadoop-qbt/39/
[Jun 1, 2016 12:41:15 AM] (kai.zheng) Revert "HADOOP-12579. Deprecate and
remove WriteableRPCEngine.
-1 overall
The following subsystems reported failures:
findbugs unit
The following subsystems are considered long running:
unit
FindBugs :
module:hadoop-common-project/hadoop-minikdc
org.apache.hadoop.minikdc.MiniKdc.stop() calls Thread.sleep() with a lock
held At MiniKdc.java:lock held At MiniKdc.java:[line 345]
Failed junit tests :
hadoop.net.TestClusterTopology
hadoop.yarn.server.resourcemanager.TestAMAuthorization
hadoop.yarn.server.resourcemanager.TestClientRMTokens
hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
hadoop.yarn.server.TestContainerManagerSecurity
hadoop.yarn.client.TestGetGroups
hadoop.yarn.client.api.impl.TestDistributedScheduling
hadoop.yarn.client.api.impl.TestAMRMProxy
hadoop.mapreduce.tools.TestCLI
hadoop.mapred.TestMiniMRChildTask
Timed out junit tests :
org.apache.hadoop.yarn.client.cli.TestYarnCLI
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
org.apache.hadoop.yarn.client.api.impl.TestYarnClient
org.apache.hadoop.yarn.client.api.impl.TestNMClient
Console output:
https://builds.apache.org/job/hadoop-qbt/39/console
Logs:
https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-compile-cc-root.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-compile-javac-root.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-checkstyle-root.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-patch-pylint.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-patch-shellcheck.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-patch-shelldocs.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out/whitespace-eol.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out/whitespace-tabs.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out/branch-findbugs-hadoop-common-project_hadoop-minikdc-warnings.html
https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-javadoc-javadoc-root.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
https://builds.apache.org/job/hadoop-qbt/39/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org
==== cut here ====
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
