[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/12980 [SPARK-15207][BUILD] Use Travis CI for Java/Scala/Python Linter and JDK7/8 compilation test ## What changes were proposed in this pull request? Currently, Java Linter is disabled due to lack of testing resources. https://github.com/apache/spark/blob/master/dev/run-tests.py#L554 This PR aims to take advantage of Travis CI to handle the following static analysis by adding a single file, `.travis.yml` without any additional burden on the existing servers. - Apache License Check - Python Linter - Scala Linter - Java Linter - JDK7/JDK8 maven compile Note that this issue does not propose to remove some of the above work items from the Jenkins. It's possible, but we need to observe the Travis CI stability for a while. The main goal of this issue is to removing committer's overhead on linter-related PRs (the original PR and the fixation PR). ## How was this patch tested? Pass the Travis CI tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dongjoon-hyun/spark SPARK-15207 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12980.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12980 commit 60638227ed3d11bcf20df877aeeda0f46477b868 Author: Dongjoon Hyun Date: 2016-05-08T01:53:17Z [SPARK-15207][BUILD] Use Travis CI for Java/Scala/Python Linter and JDK7/8 compilation test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217682754 **[Test build #58080 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58080/consoleFull)** for PR 12980 at commit [`6063822`](https://github.com/apache/spark/commit/60638227ed3d11bcf20df877aeeda0f46477b868). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217683609 Travis CI passed. It took less than 25 minutes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217683635 Hi, @rxin and @andrewor14 . I think this is the only way to re-enable Java linter. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217689275 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58080/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217689064 **[Test build #58080 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58080/consoleFull)** for PR 12980 at commit [`6063822`](https://github.com/apache/spark/commit/60638227ed3d11bcf20df877aeeda0f46477b868). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217689272 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217690316 Definitely, the failure is not related to this PR since this PR just adds `.travis.yml` file. ``` [error] (mllib/test:test) sbt.TestsFailedException: Tests unsuccessful [error] Total time: 5504 s, completed May 7, 2016 8:37:06 PM ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217702414 I don't think this hurts, except that it adds yet another place to look for test results. (PS you want to add "test-compile" as a target too). What is the reason it's disabled for Jenkins, just time? Is it really that resource intensive to check lint? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217736741 Thank you for review, @srowen . I'm not sure the root cause of that. According to the github log, it was created, [disabled (HOTFIX)](https://github.com/apache/spark/commit/6ad31e79bfe4fc1d03bd8eb4472b8bd448ee3daf) by @rxin , [re-enabled](https://github.com/apache/spark/commit/591c88c9e2a6c2e2ca84f1b66c635f198a16d112) by @rxin and [reverted](https://github.com/apache/spark/commit/4bcea1b8595424678aa6c92d66ba08c92e0fefe5) by @zsxwing historically. I've heard that from @andrewor14 that it was resource intensive, too. Anyway, two weeks ago, when I tried to re-enable that according to @rxin 's request during in the following PR, there was a functional problem on java linter on AMP clusters. [[SPARK-14868][BUILD] Enable NewLineAtEofChecker in checkstyle and fix lint-java errors](https://github.com/apache/spark/pull/12632) So, I fixed that in the following PR. Now I think that Java Linter works now in AMP cluster without any problems. [[SPARK-14867][BUILD] Remove `--force` option in `build/mvn`](https://github.com/apache/spark/pull/12631) However, in this PR, I propose to use Travis CI more actively for simple code analysis and compilation for the following main reasons. (I know that Spark uses Travis CI historically and Travis CI cannot supports Spark full testing.) - It's free. - Most GitHub community developers are more familiar with it than AMP clusters. - It will reduce the number of PR. - If we have `.travis.yml`, all the developers own forks have that. It means they will see the Travis CI result on their branches while making their PRs, **not Spark PR list.** - In the future, there is possibility to reduce the workload on AMP clusters. It means more faster processing the PR requests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217744182 **[Test build #58110 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58110/consoleFull)** for PR 12980 at commit [`690ca52`](https://github.com/apache/spark/commit/690ca526701f7ea26c2136eaca03dbaa756f15c8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217744521 Hi, @srowen . I added `test-compile`, too. You can see the result here. https://travis-ci.org/dongjoon-hyun/spark/builds/128708372 For your concern about `Yet Another Place to look for`, I think committers has no need to look into the result of Travis CI even if it fails. You can think this a kind of **speculative execution of static analysis**. Committers are able to do in the same way by just ignoring Travis CI comments. It has meaning for only the pull requester side. As you know, pull requesters do not read carefully Spark JIRA contribution guide. However, they will try to succeed on Travis CI failures definitely if they are developers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217749731 **[Test build #58110 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58110/consoleFull)** for PR 12980 at commit [`690ca52`](https://github.com/apache/spark/commit/690ca526701f7ea26c2136eaca03dbaa756f15c8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217749784 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217749785 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58110/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217805705 I don't think it's a bad thing to have some kind of Travis build running in parallel as long as people understand what it is. Maybe some extra comments in the .travis.yml file to explain its purpose? So it sounds like we could actually re-enable lint in the main build, but the concern is speed? but the Java lint checks themselves seem to take just a few seconds: ``` $ time ./dev/lint-java Using `mvn` from path: /usr/local/bin/mvn Checkstyle checks passed. real0m7.949s user0m26.882s sys 0m1.144s ``` Almost all the time in that Travis run is downloading stuff, compiling, and running the other lint checks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217908304 Yep, I did the same question before. The answer was Jenkins also do the clean build as you mentioned for Travis CI. In other words, in AMP cluster, there occurs lots of maven traffic. All CI suffers the same issue, but it's the purpose of CI. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217908555 Yeah but the Jenkins builds already have to download all that stuff to run everything else like the main build right? the additional cost of running lint-java is small? or did I misunderstand the problem? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217932005 > Currently, Java Linter is disabled due to lack of testing resources. FYI. This is not correct. The problem is `mvn checkstyle` requires that the artifacts has been published to some place. Since we publish nightly snapshots every day, `mvn checkstyle` works most of the time. However, when we upgrade the version in pom.xml or add some new projects, it requires a manual maven publish. Otherwise, `mvn checkstyle` will fail by maven resolution problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217932374 A solution is always running `mvn install` before `mvn checkstyle`. However, this wastes a lot time for sbt build. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-217958114 You're right. I maybe didn't remember the exact phrase I've read before. By the way, 'this wastes a lot time for sbt building' is effectively different from 'lack of resource'? Time/CPU/Network are all the resource. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-218029962 Hi, @zsxwing . According to your advice, I updated the PR; run `lint-java` after `mvn install`. Thank you for advice! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-218030211 **[Test build #58190 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58190/consoleFull)** for PR 12980 at commit [`ac2cf27`](https://github.com/apache/spark/commit/ac2cf27e8e428719e422e59bd9ba55a1580ee263). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15207][BUILD] Use Travis CI for Java/Sc...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12980#issuecomment-218038498 Travic CI was finished with JDK7 (30min) and JDK8 (31min) in parallel. https://travis-ci.org/dongjoon-hyun/spark/builds/129008329 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
