There are a few things going on here wrt tests. 1. I fixed up the RAT issues with a hotfix.
2. The Hive tests were actually disabled for a while accidentally. A recent fix correctly re-enabled them. Without Hive Spark tests run in about 40 minutes and with Hive it runs in 1 hour and 15 minutes, so it's a big difference. To ease things I committed a patch today that only runs the Hive tests if the change touches Spark SQL. So this should make it simpler for normal tests. We can actually generalize this to do much finer grained testing, e.g. if something in MLLib changes we don't need to re-run the streaming tests. I've added this JIRA to track it: https://issues.apache.org/jira/browse/SPARK-1455 3. Overall we've experienced more race conditions with tests recently. I noticed a few zombie test processes on Jenkins hogging up 100% of CPU so I think this has triggered several previously unseen races due to CPU contention on the test cluster. I killed them and we'll see if they crop up again. 4. Please try to keep an eye on the length of new tests that get committed. It's common to see people commit tests that e.g. sleep for several seconds or do things that take a long time. Almost always this can be avoided and usually avoiding it makes the test cleaner anyways (e.g. use proper synchronization instead of sleeping). - Patrick On Tue, Apr 15, 2014 at 9:34 AM, Mark Hamstra <m...@clearstorydata.com>wrote: > The RAT path issue is now fixed, but it appears to me that some recent > change has dramatically altered the behavior of the testing framework, so > that I am now seeing many individual tests taking more than a minute to run > and the complete test run taking a very, very long time. I expect that > this is what is causing Jenkins to now timeout repeatedly. > > > On Mon, Apr 14, 2014 at 1:32 PM, Nan Zhu <zhunanmcg...@gmail.com> wrote: > > > +1.... > > > > -- > > Nan Zhu > > > > > > On Friday, April 11, 2014 at 5:35 PM, DB Tsai wrote: > > > > > I always got > > > > ========================================================================= > > > > > > Could not find Apache license headers in the following files: > > > !????? /root/workspace/SparkPullRequestBuilder/python/metastore/db.lck > > > !????? > > > /root/workspace/SparkPullRequestBuilder/python/metastore/service.properties > > > > > > > > > Sincerely, > > > > > > DB Tsai > > > ------------------------------------------------------- > > > My Blog: https://www.dbtsai.com > > > LinkedIn: https://www.linkedin.com/in/dbtsai > > > > > > > > > > > > >
