It looks like we have come pretty close to our build time limit with Travis again. Some builds are sneaking in under 50 minutes while others are surpassing the limit and tripping the Travis timeout. This is a separate issue from the maven binary download issue https://issues.apache.org/jira/browse/METRON-2155, which we also see from time to time. I looked at one of the recent build failures from https://github.com/apache/metron/pull/1477. It's failing in the integration tests:
1803.88s = 30 min export SCRIPT="mvn surefire:test@integration-tests" time $SCRIPT 1021.49s = 17 min time mvn install -T 2C -q -DskipTests=true -Dmaven.javadoc.skip=true -B -V There's roughly 3 minutes of time not completely accounted for since the other timings listed in the log output are only a few seconds here and there, but I'm less concerned about that. Here's is the breakdown from Maven. Vanilla build without tests We don't seem to get this output detail, but the total time is 17 min. Integration tests only - uses build artifacts from the previous mvn install command [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Metron ............................................. SUCCESS [ 0.498 s] [INFO] metron-stellar ..................................... SUCCESS [ 0.180 s] [INFO] stellar-common ..................................... SUCCESS [ 17.853 s] [INFO] metron-analytics ................................... SUCCESS [ 0.026 s] [INFO] metron-maas-common ................................. SUCCESS [ 6.604 s] [INFO] metron-platform .................................... SUCCESS [ 0.029 s] [INFO] metron-zookeeper ................................... SUCCESS [ 0.031 s] [INFO] metron-test-utilities .............................. SUCCESS [ 0.323 s] [INFO] metron-integration-test ............................ SUCCESS [ 0.287 s] [INFO] metron-maas-service ................................ SUCCESS [ 58.873 s] [INFO] metron-common ...................................... SUCCESS [ 58.798 s] [INFO] metron-statistics .................................. SUCCESS [ 1.948 s] [INFO] metron-common-streaming ............................ SUCCESS [ 0.012 s] [INFO] metron-common-storm ................................ SUCCESS [ 0.038 s] [INFO] metron-hbase ....................................... SUCCESS [ 0.010 s] [INFO] metron-hbase-common ................................ SUCCESS [ 0.350 s] [INFO] metron-enrichment .................................. SUCCESS [ 0.071 s] [INFO] metron-enrichment-common ........................... SUCCESS [ 2.760 s] [INFO] metron-writer ...................................... SUCCESS [ 0.009 s] [INFO] metron-writer-common ............................... SUCCESS [ 0.046 s] [INFO] metron-writer-storm ................................ SUCCESS [ 0.156 s] [INFO] metron-storm-kafka-override ........................ SUCCESS [ 0.069 s] [INFO] metron-storm-kafka ................................. SUCCESS [ 0.066 s] [INFO] metron-profiler-common ............................. SUCCESS [ 0.042 s] [INFO] metron-profiler-client ............................. SUCCESS [ 0.053 s] [INFO] metron-profiler-storm .............................. SUCCESS [04:43 min] [INFO] metron-profiler-spark .............................. SUCCESS [02:46 min] [INFO] metron-profiler-repl ............................... SUCCESS [ 0.049 s] [INFO] metron-hbase-client ................................ SUCCESS [ 0.040 s] [INFO] metron-enrichment-storm ............................ SUCCESS [01:04 min] [INFO] metron-indexing .................................... SUCCESS [ 0.007 s] [INFO] metron-indexing-common ............................. SUCCESS [ 1.694 s] [INFO] metron-solr ........................................ SUCCESS [ 0.011 s] [INFO] metron-solr-common ................................. SUCCESS [04:40 min] [INFO] metron-indexing-storm .............................. SUCCESS [ 23.510 s] [INFO] metron-solr-storm .................................. SUCCESS [ 33.429 s] [INFO] metron-job ......................................... SUCCESS [ 0.026 s] [INFO] metron-pcap ........................................ SUCCESS [ 0.059 s] [INFO] metron-pcap-backend ................................ SUCCESS [01:26 min] [INFO] metron-data-management ............................. SUCCESS [02:55 min] [INFO] metron-parsing ..................................... SUCCESS [ 0.009 s] [INFO] metron-parsers-common .............................. SUCCESS [ 0.561 s] [INFO] metron-parsers ..................................... SUCCESS [ 0.048 s] [INFO] metron-management .................................. SUCCESS [01:24 min] [INFO] elasticsearch-shaded ............................... SUCCESS [ 0.235 s] [INFO] metron-elasticsearch ............................... SUCCESS [ 0.007 s] [INFO] metron-elasticsearch-common ........................ SUCCESS [02:11 min] [INFO] metron-elasticsearch-storm ......................... SUCCESS [ 49.759 s] [INFO] metron-parsing-storm ............................... SUCCESS [02:10 min] [INFO] metron-hbase-server ................................ SUCCESS [ 41.935 s] [INFO] metron-deployment .................................. SUCCESS [ 0.007 s] [INFO] Elasticsearch Ambari Management Pack ............... SUCCESS [ 0.007 s] [INFO] Metron Ambari Management Pack ...................... SUCCESS [ 0.034 s] [INFO] metron-contrib ..................................... SUCCESS [ 0.007 s] [INFO] metron-docker ...................................... SUCCESS [ 0.017 s] [INFO] metron-performance ................................. SUCCESS [ 0.024 s] [INFO] metron-interface ................................... SUCCESS [ 0.008 s] [INFO] metron-config ...................................... SUCCESS [ 0.042 s] [INFO] metron-alerts ...................................... SUCCESS [ 0.005 s] [INFO] metron-rest-client ................................. SUCCESS [ 0.043 s] [INFO] metron-rest ........................................ SUCCESS [01:31 min] [INFO] site-book .......................................... SUCCESS [ 0.026 s] [INFO] 3rd party Functions (just for tests) ............... SUCCESS [ 0.027 s] [INFO] stellar-zeppelin ................................... SUCCESS [ 3.782 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 30:01 min [INFO] Finished at: 2019-08-10T16:07:59+00:00 [INFO] Final Memory: 81M/686M [INFO] ------------------------------------------------------------------------ There are some parts of the build that are probably low hanging fruit, e.g. metron-data-management, metron-solr-common, metron-profiler-storm. We've already done a fair amount of work on this front by splitting up the build into 4 distinct units, but I think we're going to need to perform a deeper assessment into managing our build times on an ongoing basis. Even if we can shave a few minutes off of the integration tests now, we're very likely to run into the problem again soon as we expand our features. I'm not sure if we've tried running the integration tests in parallel or not, but I'm going to give this a shot locally. I expect this will not work reliably given that we haven't taken precautions to allow our tests to be run in parallel. Best, Mike