chiming in, as i'm the one who currently maintains the CI infrastructure... :)
+1 on not having more than one CI system... there's no way i can commit to keeping an eye on anything else other than jenkins. and i agree wholeheartedly w/michael: if it's this important, let's add it to the jenkins builds. On Mon, May 23, 2016 at 12:20 PM, Michael Armbrust <mich...@databricks.com> wrote: > We did turn on travis a few years ago, but ended up turning it off because > it was failing (I believe because of insufficient resources) which was > confusing for developers. I wouldn't be opposed to turning it on if it > provides more/faster signal, but its not obvious to me that it would. In > particular, do we know that given the rate PRs are created if we will hit > rate limits? > > Really my main feedback is, if the java linter is important we should > probably have it as part of the canonical build process. I worry about > having more than one set of CI infrastructure to maintain. > > On Mon, May 23, 2016 at 9:43 AM, Dongjoon Hyun <dongj...@apache.org> wrote: >> >> Thank you, Steve and Hyukjin. >> >> And, don't worry, Ted. >> >> Travis launches new VMs for every PR. >> >> Apache Spark repository uses the following setting. >> >> VM: Google Compute Engine >> OS: Ubuntu 14.04.3 LTS Server Edition 64bit >> CPU: ~2 CORE >> RAM: 7.5GB >> >> FYI, you can find more information about this here. >> >> >> https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments >> >> Dongjoon. >> >> >> >> On Mon, May 23, 2016 at 6:32 AM, Ted Yu <yuzhih...@gmail.com> wrote: >>> >>> Do you know if more than one PR would be verified on the same machine ? >>> >>> I wonder whether the 'mvn install' from two simultaneous PR builds may >>> have conflict. >>> >>> On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun <dongj...@apache.org> >>> wrote: >>>> >>>> Thank you for feedback. Sure, correctly, that's the reason why the >>>> current SparkPullRequestBuilder do not run `lint-java`. :-) >>>> >>>> In addition, that's the same reason why contributors are reluctant to >>>> run `lint-java` and causes breaking on JDK7 builds. >>>> >>>> Such a tedious and time-consuming job should be done by CI without human >>>> interventions. >>>> >>>> By the way, why do you think we need to wait for that? We should not >>>> wait for any CIs, we should continue our own work. >>>> >>>> My proposal isn't for making you wait to watch the result. There are two >>>> use cases I want for us to focus here. >>>> >>>> Case 1: When you make a PR to Spark PR queue. >>>> >>>> Travis CI will finish before SparkPullRequestBuilder. >>>> We will run the followings in parallel mode. >>>> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no Java >>>> Linter) >>>> 2. Travis: JDK7 + mvn build + Java Linter >>>> 3. Travis: JDK8 + mvn build + Java Linter >>>> As we know, 1 is the longest time-consuming one which have lots of >>>> works (except maven building or lint- java). You don't need to wait more >>>> in >>>> many cases. Yes, in many cases, not all the cases. >>>> >>>> >>>> Case 2: When you prepare a PR on your branch. >>>> >>>> If you are at the final commit (maybe already-squashed), just go to >>>> case 1. >>>> >>>> However, usually, we makes lots of commits locally while making >>>> preparing our PR. >>>> And, finally we squashed them into one and send a PR to Spark. >>>> I mean you can use Travis CI during preparing your PRs. >>>> Again, don't wait for Travis CI. Just push it sometime or at every >>>> commit, and continue your work. >>>> >>>> At the final stage when you finish your coding, squash your commits >>>> into one, >>>> and amend your commit title or messages, see the Travis CI. >>>> Or, you can monitor Travis CI result on status menu bar. >>>> If it shows green icon, you have nothing to do. >>>> >>>> https://docs.travis-ci.com/user/apps/ >>>> >>>> To sum up, I think we don't need to wait for any CIs. It's like an >>>> email. `Send and back to work.` >>>> >>>> Dongjoon. >>>> >>>> >>>> On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>>> >>>>> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes. >>>>> >>>>> Maybe not everyone is willing to wait that long. >>>>> >>>>> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <dongj...@apache.org> >>>>> wrote: >>>>>> >>>>>> Oh, Sure. My bad! >>>>>> >>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`. >>>>>> >>>>>> Thank you, Ted. >>>>>> >>>>>> Dongjoon. >>>>>> >>>>>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>>> >>>>>>> The following line was repeated twice: >>>>>>> >>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>>> >>>>>>> Did you intend to cover JDK 8 ? >>>>>>> >>>>>>> Cheers >>>>>>> >>>>>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <dongj...@apache.org> >>>>>>> wrote: >>>>>>>> >>>>>>>> Hi, All. >>>>>>>> >>>>>>>> I want to propose the followings. >>>>>>>> >>>>>>>> - Turn on Travis CI for Apache Spark PR queue. >>>>>>>> - Recommend this for contributors, too >>>>>>>> >>>>>>>> Currently, Spark provides Travis CI configuration file to help >>>>>>>> contributors check Scala/Java style conformance and JDK7/8 compilation >>>>>>>> easily during their preparing pull requests. Please note that it's only >>>>>>>> about static analysis. >>>>>>>> >>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>>>> Scalastyle is included in the step 'mvn install', too. >>>>>>>> >>>>>>>> Yep, if you turn on your Travis CI configuration, you can already >>>>>>>> see the results on your branches before making PR. I wrote this email >>>>>>>> to >>>>>>>> prevent more failures proactively and community-widely. >>>>>>>> >>>>>>>> For stability, I have been monitoring that for two weeks. It detects >>>>>>>> the failures or recovery on JDK7 builds or Java linter on Spark master >>>>>>>> branch correctly. The only exceptional case I observed rarely is >>>>>>>> `timeout` >>>>>>>> failure due to hangs of maven. But, as we know, it's happen in our >>>>>>>> Jenkins >>>>>>>> SparkPullRequestBuilder, too. I think we can ignore that. >>>>>>>> >>>>>>>> I'm sure that this will save much more community's efforts on the >>>>>>>> static errors by preventing them at the very early stage. But, there >>>>>>>> might >>>>>>>> be another reason not to do this. I'm wondering about your thoughts. >>>>>>>> >>>>>>>> I can make a Apache INFRA Jira issue for this if there is some >>>>>>>> consensus. >>>>>>>> >>>>>>>> Warmly, >>>>>>>> Dongjoon. >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
