The PR builder currently builds against Hadoop 2.3. - Patrick
On Fri, May 15, 2015 at 11:40 AM, Marcelo Vanzin <van...@cloudera.com> wrote: > Funny thing, since I asked this question in a PR a few minutes ago... > > Ignoring the rotation suggestion for a second, can the PR builder at least > cover hadoop 2.2? That's the actual version used to create the official > Spark artifacts for maven, and the oldest version Spark supports for YARN.. > > Kinda the same argument as the "why do we build with java 7 when we > support java 6" discussion we had recently. > > > On Fri, May 15, 2015 at 11:34 AM, Ted Yu <yuzhih...@gmail.com> wrote: > >> bq. would be prohibitive to build all configurations for every push >> >> Agreed. >> >> Can PR builder rotate testing against hadoop 2.3, 2.4, 2.6 and 2.7 (each >> test run still uses one hadoop profile) ? >> >> This way we would have some coverage for each of the major hadoop >> releases. >> >> Cheers >> >> On Fri, May 15, 2015 at 10:30 AM, Sean Owen <so...@cloudera.com> wrote: >> >>> You all are looking only at the pull request builder. It just does one >>> build to sanity-check a pull request, since that already takes 2 hours and >>> would be prohibitive to build all configurations for every push. There is a >>> different set of Jenkins jobs that periodically tests master against a lot >>> more configurations, including Hadoop 2.4. >>> >>> On Fri, May 15, 2015 at 6:02 PM, Frederick R Reiss <frre...@us.ibm.com> >>> wrote: >>> >>>> The PR builder seems to be building against Hadoop 2.3. In the log for >>>> the most recent successful build ( >>>> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32805/consoleFull >>>> ) I see: >>>> >>>> >>>> ========================================================================= >>>> Building Spark >>>> >>>> ========================================================================= >>>> [info] Compile with Hive 0.13.1 >>>> [info] Building Spark with these arguments: -Pyarn -Phadoop-2.3 >>>> -Dhadoop.version=2.3.0 -Pkinesis-asl -Phive -Phive-thriftserver >>>> ... >>>> >>>> ========================================================================= >>>> Running Spark unit tests >>>> >>>> ========================================================================= >>>> [info] Running Spark tests with these arguments: -Pyarn -Phadoop-2.3 >>>> -Dhadoop.version=2.3.0 -Pkinesis-asl test >>>> >>>> Is anyone testing individual pull requests against Hadoop 2.4 or 2.6 >>>> before the code is declared "clean"? >>>> >>>> Fred >>>> >>>> [image: Inactive hide details for Ted Yu ---05/15/2015 09:29:09 >>>> AM---Jenkins build against hadoop 2.4 has been unstable recently: https]Ted >>>> Yu ---05/15/2015 09:29:09 AM---Jenkins build against hadoop 2.4 has been >>>> unstable recently: https://amplab.cs.berkeley.edu/jenkins/ >>>> >>>> From: Ted Yu <yuzhih...@gmail.com> >>>> To: Andrew Or <and...@databricks.com> >>>> Cc: "dev@spark.apache.org" <dev@spark.apache.org> >>>> Date: 05/15/2015 09:29 AM >>>> Subject: Re: Recent Spark test failures >>>> ------------------------------ >>>> >>>> >>>> >>>> Jenkins build against hadoop 2.4 has been unstable recently: >>>> >>>> *https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/* >>>> <https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/> >>>> >>>> I haven't found the test which hung / failed in recent Jenkins builds. >>>> >>>> But PR builder has several green builds lately: >>>> *https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/* >>>> <https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/> >>>> >>>> Maybe PR builder doesn't build against hadoop 2.4 ? >>>> >>>> Cheers >>>> >>>> On Mon, May 11, 2015 at 1:11 PM, Ted Yu <*yuzhih...@gmail.com* >>>> <yuzhih...@gmail.com>> wrote: >>>> >>>> Makes sense. >>>> >>>> Having high determinism in these tests would make Jenkins build >>>> stable. >>>> >>>> >>>> On Mon, May 11, 2015 at 1:08 PM, Andrew Or <*and...@databricks.com* >>>> <and...@databricks.com>> wrote: >>>> Hi Ted, >>>> >>>> Yes, those two options can be useful, but in general I think the >>>> standard to set is that tests should never fail. It's actually the >>>> worst if >>>> tests fail sometimes but not others, because we can't reproduce them >>>> deterministically. Using -M and -A actually tolerates flaky tests to >>>> a >>>> certain extent, and I would prefer to instead increase the >>>> determinism in >>>> these tests. >>>> >>>> -Andrew >>>> >>>> 2015-05-08 17:56 GMT-07:00 Ted Yu <*yuzhih...@gmail.com* >>>> <yuzhih...@gmail.com>>: >>>> Andrew: >>>> Do you think the -M and -A options described here can be used >>>> in test runs ? >>>> *http://scalatest.org/user_guide/using_the_runner* >>>> <http://scalatest.org/user_guide/using_the_runner> >>>> >>>> Cheers >>>> >>>> On Wed, May 6, 2015 at 5:41 PM, Andrew Or < >>>> *and...@databricks.com* <and...@databricks.com>> wrote: >>>> Dear all, >>>> >>>> I'm sure you have all noticed that the Spark tests have >>>> been fairly >>>> unstable recently. I wanted to share a tool that I use to >>>> track which tests >>>> have been failing most often in order to prioritize fixing >>>> these flaky >>>> tests. >>>> >>>> Here is an output of the tool. This spreadsheet reports >>>> the top 10 failed >>>> tests this week (ending yesterday 5/5): >>>> >>>> >>>> *https://docs.google.com/spreadsheets/d/1Iv_UDaTFGTMad1sOQ_s4ddWr6KD3PuFIHmTSzL7LSb4* >>>> >>>> <https://docs.google.com/spreadsheets/d/1Iv_UDaTFGTMad1sOQ_s4ddWr6KD3PuFIHmTSzL7LSb4> >>>> >>>> It is produced by a small project: >>>> *https://github.com/andrewor14/spark-test-failures* >>>> <https://github.com/andrewor14/spark-test-failures> >>>> >>>> I have been filing JIRAs on flaky tests based on this >>>> tool. Hopefully we >>>> can collectively stabilize the build a little more as we >>>> near the release >>>> for Spark 1.4. >>>> >>>> -Andrew >>>> >>>> >>>> >>>> >>> >> > > > -- > Marcelo >
