Thank you, Shane! I really hope that SparkPullRequestBuilder handle them if possible.
Dongjoon. On Mon, May 23, 2016 at 1:24 PM, Dongjoon Hyun <dongj...@apache.org> wrote: > Thank you for your opinion! > > Sure. I know that history and totally agree with all your concerns. > I indeed has hesitated about sending this kind of suggestion for a while. > > If Travis CI cannot handle those simple jobs at this time again, > we must turn off from Spark PR queue. > We can see the result quickly in one or two days. > To turn on/off, Spark have nothing to do. INFRA team will do that. > > In fact, the goal is not about using another CI (like Travis), it is about > preventing the followings. > > 1. JDK7 compilation errors. (Recently, 2 days ago and 5 days ago) > 2. Java static errors. (Not critical but more frequently.) > 3. Maven installation errors. (A month ago, it's reported in this mailing > list.) > > Scala 2.10 compilation errors are fixed nearly instantly. But, 1~3 were > not. > If SparkPullRequestBuilder can do the above 1~3, that's the best for us. > Do you think it is possible in some ways? > > By the way, as of today, Spark has 724 Java files and 96762 lines (without > comment/blank). > It's about 1/3 of Scala code. It's not small. > -------------------------------------------------------------------------- > Language files blank comment code > -------------------------------------------------------------------------- > Scala 2368 63578 124904 322518 > Java 724 18569 23445 > 96762 > > Dongjoon. > > > > On Mon, May 23, 2016 at 12:20 PM, Michael Armbrust <mich...@databricks.com > > wrote: > >> We did turn on travis a few years ago, but ended up turning it off >> because it was failing (I believe because of insufficient resources) which >> was confusing for developers. I wouldn't be opposed to turning it on if it >> provides more/faster signal, but its not obvious to me that it would. In >> particular, do we know that given the rate PRs are created if we will hit >> rate limits? >> >> Really my main feedback is, if the java linter is important we should >> probably have it as part of the canonical build process. I worry about >> having more than one set of CI infrastructure to maintain. >> >> On Mon, May 23, 2016 at 9:43 AM, Dongjoon Hyun <dongj...@apache.org> >> wrote: >> >>> Thank you, Steve and Hyukjin. >>> >>> And, don't worry, Ted. >>> >>> Travis launches new VMs for every PR. >>> >>> Apache Spark repository uses the following setting. >>> >>> VM: Google Compute Engine >>> OS: Ubuntu 14.04.3 LTS Server Edition 64bit >>> CPU: ~2 CORE >>> RAM: 7.5GB >>> >>> FYI, you can find more information about this here. >>> >>> >>> https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments >>> >>> Dongjoon. >>> >>> >>> >>> On Mon, May 23, 2016 at 6:32 AM, Ted Yu <yuzhih...@gmail.com> wrote: >>> >>>> Do you know if more than one PR would be verified on the same machine ? >>>> >>>> I wonder whether the 'mvn install' from two simultaneous PR builds may >>>> have conflict. >>>> >>>> On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun <dongj...@apache.org> >>>> wrote: >>>> >>>>> Thank you for feedback. Sure, correctly, that's the reason why the >>>>> current SparkPullRequestBuilder do not run `lint-java`. :-) >>>>> >>>>> In addition, that's the same reason why contributors are reluctant to >>>>> run `lint-java` and causes breaking on JDK7 builds. >>>>> >>>>> Such a tedious and time-consuming job should be done by CI without >>>>> human interventions. >>>>> >>>>> By the way, why do you think we need to wait for that? We should not >>>>> wait for any CIs, we should continue our own work. >>>>> >>>>> My proposal isn't for making you wait to watch the result. There are >>>>> two use cases I want for us to focus here. >>>>> >>>>> Case 1: When you make a PR to Spark PR queue. >>>>> >>>>> Travis CI will finish before SparkPullRequestBuilder. >>>>> We will run the followings in parallel mode. >>>>> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no >>>>> Java Linter) >>>>> 2. Travis: JDK7 + mvn build + Java Linter >>>>> 3. Travis: JDK8 + mvn build + Java Linter >>>>> As we know, 1 is the longest time-consuming one which have lots >>>>> of works (except maven building or lint- java). You don't need to wait >>>>> more in many cases. Yes, in many cases, not all the cases. >>>>> >>>>> >>>>> Case 2: When you prepare a PR on your branch. >>>>> >>>>> If you are at the final commit (maybe already-squashed), just go >>>>> to case 1. >>>>> >>>>> However, usually, we makes lots of commits locally while making >>>>> preparing our PR. >>>>> And, finally we squashed them into one and send a PR to Spark. >>>>> I mean you can use Travis CI during preparing your PRs. >>>>> Again, don't wait for Travis CI. Just push it sometime or at every >>>>> commit, and continue your work. >>>>> >>>>> At the final stage when you finish your coding, squash your >>>>> commits into one, >>>>> and amend your commit title or messages, see the Travis CI. >>>>> Or, you can monitor Travis CI result on status menu bar. >>>>> If it shows green icon, you have nothing to do. >>>>> >>>>> https://docs.travis-ci.com/user/apps/ >>>>> >>>>> To sum up, I think we don't need to wait for any CIs. It's like an >>>>> email. `Send and back to work.` >>>>> >>>>> Dongjoon. >>>>> >>>>> >>>>> On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>>> >>>>>> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes. >>>>>> >>>>>> Maybe not everyone is willing to wait that long. >>>>>> >>>>>> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <dongj...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> Oh, Sure. My bad! >>>>>>> >>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`. >>>>>>> >>>>>>> Thank you, Ted. >>>>>>> >>>>>>> Dongjoon. >>>>>>> >>>>>>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>>> >>>>>>>> The following line was repeated twice: >>>>>>>> >>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>>>> >>>>>>>> Did you intend to cover JDK 8 ? >>>>>>>> >>>>>>>> Cheers >>>>>>>> >>>>>>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <dongj...@apache.org >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi, All. >>>>>>>>> >>>>>>>>> I want to propose the followings. >>>>>>>>> >>>>>>>>> - Turn on Travis CI for Apache Spark PR queue. >>>>>>>>> - Recommend this for contributors, too >>>>>>>>> >>>>>>>>> Currently, Spark provides Travis CI configuration file to help >>>>>>>>> contributors check Scala/Java style conformance and JDK7/8 compilation >>>>>>>>> easily during their preparing pull requests. Please note that it's >>>>>>>>> only >>>>>>>>> about static analysis. >>>>>>>>> >>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>>>>> Scalastyle is included in the step 'mvn install', too. >>>>>>>>> >>>>>>>>> Yep, if you turn on your Travis CI configuration, you can already >>>>>>>>> see the results on your branches before making PR. I wrote this email >>>>>>>>> to >>>>>>>>> prevent more failures proactively and community-widely. >>>>>>>>> >>>>>>>>> For stability, I have been monitoring that for two weeks. It >>>>>>>>> detects the failures or recovery on JDK7 builds or Java linter on >>>>>>>>> Spark >>>>>>>>> master branch correctly. The only exceptional case I observed rarely >>>>>>>>> is >>>>>>>>> `timeout` failure due to hangs of maven. But, as we know, it's happen >>>>>>>>> in >>>>>>>>> our Jenkins SparkPullRequestBuilder, too. I think we can ignore that. >>>>>>>>> >>>>>>>>> I'm sure that this will save much more community's efforts on the >>>>>>>>> static errors by preventing them at the very early stage. But, there >>>>>>>>> might >>>>>>>>> be another reason not to do this. I'm wondering about your thoughts. >>>>>>>>> >>>>>>>>> I can make a Apache INFRA Jira issue for this if there is some >>>>>>>>> consensus. >>>>>>>>> >>>>>>>>> Warmly, >>>>>>>>> Dongjoon. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
