Repository: spark-website Updated Branches: refs/heads/asf-site 39838046c -> 8b27c470c
added section about mima Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/8b27c470 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/8b27c470 Diff: http://git-wip-us.apache.org/repos/asf/spark-website/diff/8b27c470 Branch: refs/heads/asf-site Commit: 8b27c470c32b30f6bddafd4cdec2b75b5b975fb6 Parents: 3983804 Author: Benjamin Fradet <benjamin.fra...@gmail.com> Authored: Sat Mar 25 21:48:40 2017 +0000 Committer: Benjamin Fradet <benjamin.fra...@gmail.com> Committed: Sun Mar 26 18:14:45 2017 +0100 ---------------------------------------------------------------------- developer-tools.md | 56 +++++++++++++++++++++++++++++++++++++++++- site/developer-tools.html | 51 +++++++++++++++++++++++++++++++++++++- 2 files changed, 105 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark-website/blob/8b27c470/developer-tools.md ---------------------------------------------------------------------- diff --git a/developer-tools.md b/developer-tools.md index 88f3f36..547d8aa 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -111,6 +111,60 @@ To run individual Java tests, you can use the `-Dtest` flag: build/mvn test -DwildcardSuites=none -Dtest=org.apache.spark.streaming.JavaAPISuite test ``` +<h3>Binary compatibility</h3> + +To ensure binary compatibility, Spark uses [MiMa](https://github.com/typesafehub/migration-manager). + +<h4>Ensuring binary compatibility</h4> + +When working on an issue, it's always a good idea to check that your changes do +not introduce binary incompatibilities before opening a pull request. + +You can do so by running the following command: + +``` +$ dev/mima +``` + +A binary incompatibility reported by MiMa might look like the following: + +``` +[error] method this(org.apache.spark.sql.Dataset)Unit in class org.apache.spark.SomeClass does not have a correspondent in current version +[error] filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.SomeClass.this") +``` + +If you open a pull request containing binary incompatibilities anyway, Jenkins +will remind you by failing the test build with the following message: + +``` +Test build #xx has finished for PR yy at commit ffffff. + + This patch fails MiMa tests. + This patch merges cleanly. + This patch adds no public classes. +``` + +<h4>Solving a binary incompatibility</h4> + +If you believe that your binary incompatibilies are justified or that MiMa +reported false positives (e.g. the reported binary incompatibilities are about a +non-user facing API), you can filter them out by adding an exclusion in +[project/MimaExcludes.scala](https://github.com/apache/spark/blob/master/project/MimaExcludes.scala) +containing what was suggested by the MiMa report and a comment containing the +JIRA number of the issue you're working on as well as its title. + +For the problem described above, we might add the following: + +{% highlight scala %} +// [SPARK-zz][CORE] Fix an issue +ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.SomeClass.this") +{% endhighlight %} + +Otherwise, you will have to resolve those incompatibilies before opening or +updating your pull request. Usually, the problems reported by MiMa are +self-explanatory and revolve around missing members (methods or fields) that +you will have to add back in order to maintain binary compatibility. + <h3>Checking Out Pull Requests</h3> Git provides a mechanism for fetching remote pull requests into your own local repository. @@ -181,7 +235,7 @@ It is due to an incorrect Scala library in the classpath. To fix it: - Remove `scala-library-2.10.4.jar - lib_managed\jars` In the event of "Could not find resource path for Web UI: org/apache/spark/ui/static", -it's due to a classpath issue (some classes were probably not compiled). To fix this, it +it's due to a classpath issue (some classes were probably not compiled). To fix this, it is sufficient to run a test from the command line: ``` http://git-wip-us.apache.org/repos/asf/spark-website/blob/8b27c470/site/developer-tools.html ---------------------------------------------------------------------- diff --git a/site/developer-tools.html b/site/developer-tools.html index d09815d..a44bfde 100644 --- a/site/developer-tools.html +++ b/site/developer-tools.html @@ -287,6 +287,55 @@ restart whenever <code>build/mvn</code> is called.</p> <pre><code>build/mvn test -DwildcardSuites=none -Dtest=org.apache.spark.streaming.JavaAPISuite test </code></pre> +<h3>Binary compatibility</h3> + +<p>To ensure binary compatibility, Spark uses <a href="https://github.com/typesafehub/migration-manager">MiMa</a>.</p> + +<h4>Ensuring binary compatibility</h4> + +<p>When working on an issue, it’s always a good idea to check that your changes do +not introduce binary incompatibilities before opening a pull request.</p> + +<p>You can do so by running the following command:</p> + +<pre><code>$ dev/mima +</code></pre> + +<p>A binary incompatibility reported by MiMa might look like the following:</p> + +<pre><code>[error] method this(org.apache.spark.sql.Dataset)Unit in class org.apache.spark.SomeClass does not have a correspondent in current version +[error] filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.SomeClass.this") +</code></pre> + +<p>If you open a pull request containing binary incompatibilities anyway, Jenkins +will remind you by failing the test build with the following message:</p> + +<pre><code>Test build #xx has finished for PR yy at commit ffffff. + + This patch fails MiMa tests. + This patch merges cleanly. + This patch adds no public classes. +</code></pre> + +<h4>Solving a binary incompatibility</h4> + +<p>If you believe that your binary incompatibilies are justified or that MiMa +reported false positives (e.g. the reported binary incompatibilities are about a +non-user facing API), you can filter them out by adding an exclusion in +<a href="https://github.com/apache/spark/blob/master/project/MimaExcludes.scala">project/MimaExcludes.scala</a> +containing what was suggested by the MiMa report and a comment containing the +JIRA number of the issue you’re working on as well as its title.</p> + +<p>For the problem described above, we might add the following:</p> + +<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="c1">// [SPARK-zz][CORE] Fix an issue</span> +<span class="nc">ProblemFilters</span><span class="o">.</span><span class="n">exclude</span><span class="o">[</span><span class="kt">DirectMissingMethodProblem</span><span class="o">](</span><span class="s">"org.apache.spark.SomeClass.this"</span><span class="o">)</span></code></pre></figure> + +<p>Otherwise, you will have to resolve those incompatibilies before opening or +updating your pull request. Usually, the problems reported by MiMa are +self-explanatory and revolve around missing members (methods or fields) that +you will have to add back in order to maintain binary compatibility.</p> + <h3>Checking Out Pull Requests</h3> <p>Git provides a mechanism for fetching remote pull requests into your own local repository. @@ -353,7 +402,7 @@ java.lang.NullPointerException </ul> <p>In the event of “Could not find resource path for Web UI: org/apache/spark/ui/static”, -it’s due to a classpath issue (some classes were probably not compiled). To fix this, it +it’s due to a classpath issue (some classes were probably not compiled). To fix this, it is sufficient to run a test from the command line:</p> <pre><code>build/sbt "test-only org.apache.spark.rdd.SortingSuite" --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org