Port wiki Useful Developer Tools and Profiling Spark Apps to /developer-tools.html; port Spark Versioning Policy and main wiki to /versioning-policy.html; port Preparing Spark Releases to /release-process.html; rearrange menu with new Developer menu
Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/cf21826b Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/cf21826b Diff: http://git-wip-us.apache.org/repos/asf/spark-website/diff/cf21826b Branch: refs/heads/asf-site Commit: cf21826beab2b83cca0028c555e1008ae2f2ed93 Parents: 0744e8f Author: Sean Owen <so...@cloudera.com> Authored: Tue Nov 22 14:21:33 2016 +0000 Committer: Sean Owen <so...@cloudera.com> Committed: Tue Nov 22 14:38:50 2016 +0000 ---------------------------------------------------------------------- _layouts/global.html | 18 +- developer-tools.md | 287 +++++++++++ documentation.md | 7 - downloads.md | 2 +- release-process.md | 263 ++++++++++ site/committers.html | 18 +- site/community.html | 18 +- site/contributing.html | 18 +- site/developer-tools.html | 494 +++++++++++++++++++ site/documentation.html | 25 +- site/downloads.html | 20 +- site/examples.html | 18 +- site/faq.html | 18 +- site/graphx/index.html | 18 +- site/index.html | 18 +- site/mailing-lists.html | 18 +- site/mllib/index.html | 18 +- site/news/amp-camp-2013-registration-ope.html | 18 +- .../news/announcing-the-first-spark-summit.html | 18 +- .../news/fourth-spark-screencast-published.html | 18 +- site/news/index.html | 18 +- site/news/nsdi-paper.html | 18 +- site/news/one-month-to-spark-summit-2015.html | 18 +- .../proposals-open-for-spark-summit-east.html | 18 +- ...registration-open-for-spark-summit-east.html | 18 +- .../news/run-spark-and-shark-on-amazon-emr.html | 18 +- site/news/spark-0-6-1-and-0-5-2-released.html | 18 +- site/news/spark-0-6-2-released.html | 18 +- site/news/spark-0-7-0-released.html | 18 +- site/news/spark-0-7-2-released.html | 18 +- site/news/spark-0-7-3-released.html | 18 +- site/news/spark-0-8-0-released.html | 18 +- site/news/spark-0-8-1-released.html | 18 +- site/news/spark-0-9-0-released.html | 18 +- site/news/spark-0-9-1-released.html | 18 +- site/news/spark-0-9-2-released.html | 18 +- site/news/spark-1-0-0-released.html | 18 +- site/news/spark-1-0-1-released.html | 18 +- site/news/spark-1-0-2-released.html | 18 +- site/news/spark-1-1-0-released.html | 18 +- site/news/spark-1-1-1-released.html | 18 +- site/news/spark-1-2-0-released.html | 18 +- site/news/spark-1-2-1-released.html | 18 +- site/news/spark-1-2-2-released.html | 18 +- site/news/spark-1-3-0-released.html | 18 +- site/news/spark-1-4-0-released.html | 18 +- site/news/spark-1-4-1-released.html | 18 +- site/news/spark-1-5-0-released.html | 18 +- site/news/spark-1-5-1-released.html | 18 +- site/news/spark-1-5-2-released.html | 18 +- site/news/spark-1-6-0-released.html | 18 +- site/news/spark-1-6-1-released.html | 18 +- site/news/spark-1-6-2-released.html | 18 +- site/news/spark-1-6-3-released.html | 18 +- site/news/spark-2-0-0-released.html | 18 +- site/news/spark-2-0-1-released.html | 18 +- site/news/spark-2-0-2-released.html | 18 +- site/news/spark-2.0.0-preview.html | 18 +- .../spark-accepted-into-apache-incubator.html | 18 +- site/news/spark-and-shark-in-the-news.html | 18 +- site/news/spark-becomes-tlp.html | 18 +- site/news/spark-featured-in-wired.html | 18 +- .../spark-mailing-lists-moving-to-apache.html | 18 +- site/news/spark-meetups.html | 18 +- site/news/spark-screencasts-published.html | 18 +- site/news/spark-summit-2013-is-a-wrap.html | 18 +- site/news/spark-summit-2014-videos-posted.html | 18 +- site/news/spark-summit-2015-videos-posted.html | 18 +- site/news/spark-summit-agenda-posted.html | 18 +- .../spark-summit-east-2015-videos-posted.html | 18 +- .../spark-summit-east-2016-cfp-closing.html | 18 +- site/news/spark-summit-east-agenda-posted.html | 18 +- .../news/spark-summit-europe-agenda-posted.html | 18 +- site/news/spark-summit-europe.html | 18 +- .../spark-summit-june-2016-agenda-posted.html | 18 +- site/news/spark-tips-from-quantifind.html | 18 +- .../spark-user-survey-and-powered-by-page.html | 18 +- site/news/spark-version-0-6-0-released.html | 18 +- .../spark-wins-cloudsort-100tb-benchmark.html | 18 +- ...-wins-daytona-gray-sort-100tb-benchmark.html | 18 +- .../strata-exercises-now-available-online.html | 18 +- .../news/submit-talks-to-spark-summit-2014.html | 18 +- .../news/submit-talks-to-spark-summit-2016.html | 18 +- .../submit-talks-to-spark-summit-east-2016.html | 18 +- .../submit-talks-to-spark-summit-eu-2016.html | 18 +- site/news/two-weeks-to-spark-summit-2014.html | 18 +- ...deo-from-first-spark-development-meetup.html | 18 +- site/powered-by.html | 18 +- site/release-process.html | 475 ++++++++++++++++++ site/releases/spark-release-0-3.html | 18 +- site/releases/spark-release-0-5-0.html | 18 +- site/releases/spark-release-0-5-1.html | 18 +- site/releases/spark-release-0-5-2.html | 18 +- site/releases/spark-release-0-6-0.html | 18 +- site/releases/spark-release-0-6-1.html | 18 +- site/releases/spark-release-0-6-2.html | 18 +- site/releases/spark-release-0-7-0.html | 18 +- site/releases/spark-release-0-7-2.html | 18 +- site/releases/spark-release-0-7-3.html | 18 +- site/releases/spark-release-0-8-0.html | 18 +- site/releases/spark-release-0-8-1.html | 18 +- site/releases/spark-release-0-9-0.html | 18 +- site/releases/spark-release-0-9-1.html | 18 +- site/releases/spark-release-0-9-2.html | 18 +- site/releases/spark-release-1-0-0.html | 18 +- site/releases/spark-release-1-0-1.html | 18 +- site/releases/spark-release-1-0-2.html | 18 +- site/releases/spark-release-1-1-0.html | 18 +- site/releases/spark-release-1-1-1.html | 18 +- site/releases/spark-release-1-2-0.html | 18 +- site/releases/spark-release-1-2-1.html | 18 +- site/releases/spark-release-1-2-2.html | 18 +- site/releases/spark-release-1-3-0.html | 18 +- site/releases/spark-release-1-3-1.html | 18 +- site/releases/spark-release-1-4-0.html | 18 +- site/releases/spark-release-1-4-1.html | 18 +- site/releases/spark-release-1-5-0.html | 18 +- site/releases/spark-release-1-5-1.html | 18 +- site/releases/spark-release-1-5-2.html | 18 +- site/releases/spark-release-1-6-0.html | 18 +- site/releases/spark-release-1-6-1.html | 18 +- site/releases/spark-release-1-6-2.html | 18 +- site/releases/spark-release-1-6-3.html | 18 +- site/releases/spark-release-2-0-0.html | 18 +- site/releases/spark-release-2-0-1.html | 18 +- site/releases/spark-release-2-0-2.html | 18 +- site/research.html | 18 +- site/screencasts/1-first-steps-with-spark.html | 18 +- .../2-spark-documentation-overview.html | 18 +- .../3-transformations-and-caching.html | 18 +- .../4-a-standalone-job-in-spark.html | 18 +- site/screencasts/index.html | 18 +- site/sitemap.xml | 12 + site/sql/index.html | 18 +- site/streaming/index.html | 18 +- site/third-party-projects.html | 18 +- site/trademarks.html | 18 +- site/versioning-policy.html | 320 ++++++++++++ versioning-policy.md | 75 +++ 139 files changed, 3618 insertions(+), 666 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/_layouts/global.html ---------------------------------------------------------------------- diff --git a/_layouts/global.html b/_layouts/global.html index 662fb86..0fbfe5a 100644 --- a/_layouts/global.html +++ b/_layouts/global.html @@ -123,24 +123,32 @@ <ul class="dropdown-menu"> <li><a href="{{site.baseurl}}/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="{{site.baseurl}}/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="{{site.baseurl}}/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="{{site.baseurl}}/examples.html">Examples</a></li> <li class="dropdown"> - <a href="{{site.baseurl}}/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="{{site.baseurl}}/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="{{site.baseurl}}/community.html">Mailing Lists & Resources</a></li> <li><a href="{{site.baseurl}}/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="{{site.baseurl}}/community.html#events">Events and Meetups</a></li> - <li><a href="{{site.baseurl}}/community.html#history">Project History</a></li> <li><a href="{{site.baseurl}}/powered-by.html">Powered By</a></li> <li><a href="{{site.baseurl}}/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="{{site.baseurl}}/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="{{site.baseurl}}/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="{{site.baseurl}}/versioning-policy.html">Versioning Policy</a></li> + <li><a href="{{site.baseurl}}/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/developer-tools.md ---------------------------------------------------------------------- diff --git a/developer-tools.md b/developer-tools.md new file mode 100644 index 0000000..77d225f --- /dev/null +++ b/developer-tools.md @@ -0,0 +1,287 @@ +--- +layout: global +title: Useful Developer Tools +type: "page singular" +navigation: + weight: 5 + show: true +--- + +<h2>Useful Developer Tools</h2> + +<h3>Reducing Build Times</h3> + +Spark's default build strategy is to assemble a jar including all of its dependencies. This can +be cumbersome when doing iterative development. When developing locally, it is possible to create +an assembly jar including all of Spark's dependencies and then re-package only Spark itself +when making changes. + +``` +$ build/sbt clean package +$ ./bin/spark-shell +$ export SPARK_PREPEND_CLASSES=true +$ ./bin/spark-shell # Now it's using compiled classes +# ... do some local development ... # +$ build/sbt compile +# ... do some local development ... # +$ build/sbt compile +$ unset SPARK_PREPEND_CLASSES +$ ./bin/spark-shell + +# You can also use ~ to let sbt do incremental builds on file changes without running a new sbt session every time +$ build/sbt ~compile +``` + +<h3>Checking Out Pull Requests</h3> + +Git provides a mechanism for fetching remote pull requests into your own local repository. +This is useful when reviewing code or testing patches locally. If you haven't yet cloned the +Spark Git repository, use the following command: + +``` +$ git clone https://github.com/apache/spark.git +$ cd spark +``` + +To enable this feature you'll need to configure the git remote repository to fetch pull request +data. Do this by modifying the `.git/config` file inside of your Spark directory. The remote may +not be named "origin" if you've named it something else: + +``` +[remote "origin"] + url = g...@github.com:apache/spark.git + fetch = +refs/heads/*:refs/remotes/origin/* + fetch = +refs/pull/*/head:refs/remotes/origin/pr/* # Add this line +``` + +Once you've done this you can fetch remote pull requests + +``` +# Fetch remote pull requests +$ git fetch origin +# Checkout a remote pull request +$ git checkout origin/pr/112 +# Create a local branch from a remote pull request +$ git checkout origin/pr/112 -b new-branch +``` + +<h3>Generating Dependency Graphs</h3> + +``` +$ # sbt +$ build/sbt dependency-tree + +$ # Maven +$ build/mvn -DskipTests install +$ build/mvn dependency:tree +``` + +<a name="individual-tests"></a> +<h3>Running Build Targets For Individual Projects</h3> + +``` +$ # sbt +$ build/sbt package +$ # Maven +$ build/mvn package -DskipTests -pl assembly +``` + +<h3>ScalaTest Issues</h3> + +If the following error occurs when running ScalaTest + +``` +An internal error occurred during: "Launching XYZSuite.scala". +java.lang.NullPointerException +``` +It is due to an incorrect Scala library in the classpath. To fix it: + +- Right click on project +- Select `Build Path | Configure Build Path` +- `Add Library | Scala Library` +- Remove `scala-library-2.10.4.jar - lib_managed\jars` + +In the event of "Could not find resource path for Web UI: org/apache/spark/ui/static", +it's due to a classpath issue (some classes were probably not compiled). To fix this, it +sufficient to run a test from the command line: + +``` +build/sbt "test-only org.apache.spark.rdd.SortingSuite" +``` + +<h3>Running Different Test Permutations on Jenkins</h3> + +When running tests for a pull request on Jenkins, you can add special phrases to the title of +your pull request to change testing behavior. This includes: + +- `[test-maven]` - signals to test the pull request using maven +- `[test-hadoop1.0]` - signals to test using Spark's Hadoop 1.0 profile (other options include +Hadoop 2.0, 2.2, and 2.3) + +<h3>Organizing Imports</h3> + +You can use a <a href="https://plugins.jetbrains.com/plugin/7350";>IntelliJ Imports Organizer</a> +from Aaron Davidson to help you organize the imports in +your code. It can be configured to match the import ordering from the style guide. + +<h3>IDE Setup</h3> + +<h4>IntelliJ</h4> + +While many of the Spark developers use SBT or Maven on the command line, the most common IDE we +use is IntelliJ IDEA. You can get the community edition for free (Apache committers can get +free IntelliJ Ultimate Edition licenses) and install the JetBrains Scala plugin from `Preferences > Plugins`. + +To create a Spark project for IntelliJ: + +- Download IntelliJ and install the +<a href="https://confluence.jetbrains.com/display/SCA/Scala+Plugin+for+IntelliJ+IDEA";>Scala plug-in for IntelliJ</a>. +- Go to `File -> Import Project`, locate the spark source directory, and select "Maven Project". +- In the Import wizard, it's fine to leave settings at their default. However it is usually useful +to enable "Import Maven projects automatically", since changes to the project structure will +automatically update the IntelliJ project. +- As documented in <a href="http://spark.apache.org/docs/latest/building-spark.html";>Building Spark</a>, +some build configurations require specific profiles to be +enabled. The same profiles that are enabled with `-P[profile name]` above may be enabled on the +Profiles screen in the Import wizard. For example, if developing for Hadoop 2.4 with YARN support, +enable profiles yarn and hadoop-2.4. These selections can be changed later by accessing the +"Maven Projects" tool window from the View menu, and expanding the Profiles section. + +Other tips: + +- "Rebuild Project" can fail the first time the project is compiled, because generate source files +are not automatically generated. Try clicking the "Generate Sources and Update Folders For All +Projects" button in the "Maven Projects" tool window to manually generate these sources. +- Some of the modules have pluggable source directories based on Maven profiles (i.e. to support +both Scala 2.11 and 2.10 or to allow cross building against different versions of Hive). In some +cases IntelliJ's does not correctly detect use of the maven-build-plugin to add source directories. +In these cases, you may need to add source locations explicitly to compile the entire project. If +so, open the "Project Settings" and select "Modules". Based on your selected Maven profiles, you +may need to add source folders to the following modules: + - spark-hive: add v0.13.1/src/main/scala + - spark-streaming-flume-sink: add target\scala-2.10\src_managed\main\compiled_avro +- Compilation may fail with an error like "scalac: bad option: +-P:/home/jakub/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar". +If so, go to Preferences > Build, Execution, Deployment > Scala Compiler and clear the "Additional +compiler options" field. It will work then although the option will come back when the project +reimports. If you try to build any of the projects using quasiquotes (eg., sql) then you will +need to make that jar a compiler plugin (just below "Additional compiler options"). +Otherwise you will see errors like: +``` +/Users/irashid/github/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala +Error:(147, 9) value q is not a member of StringContext + Note: implicit class Evaluate2 is not applicable here because it comes after the application point and it lacks an explicit result type + q""" + ^ +``` + +<h4>Eclipse</h4> + +Eclipse can be used to develop and test Spark. The following configuration is known to work: + +- Eclipse Juno +- <a href="http://scala-ide.org/";>Scala IDE 4.0</a> +- Scala Test + +The easiest way is to download the Scala IDE bundle from the Scala IDE download page. It comes +pre-installed with ScalaTest. Alternatively, use the Scala IDE update site or Eclipse Marketplace. + +SBT can create Eclipse `.project` and `.classpath` files. To create these files for each Spark sub +project, use this command: + +``` +sbt/sbt eclipse +``` + +To import a specific project, e.g. spark-core, select `File | Import | Existing Projects` into +Workspace. Do not select "Copy projects into workspace". + +If you want to develop on Scala 2.10 you need to configure a Scala installation for the +exact Scala version thatâs used to compile Spark. + Since Scala IDE bundles the latest versions (2.10.5 and 2.11.8 at this point), you need to add one +in `Eclipse Preferences -> Scala -> Installations` by pointing to the `lib/` directory of your +Scala 2.10.5 distribution. Once this is done, select all Spark projects and right-click, +choose `Scala -> Set Scala Installation` and point to the 2.10.5 installation. +This should clear all errors about invalid cross-compiled libraries. A clean build should succeed now. + +ScalaTest can execute unit tests by right clicking a source file and selecting `Run As | Scala Test`. + +If Java memory errors occur, it might be necessary to increase the settings in `eclipse.ini` +in the Eclipse install directory. Increase the following setting as needed: + +``` +--launcher.XXMaxPermSize +256M +``` + +<a name="nightly-builds"></a> +<h3>Nightly Builds</h3> + +Packages are built regularly off of Spark's master branch and release branches. These provide +Spark developers access to the bleeding-edge of Spark master or the most recent fixes not yet +incorporated into a maintenance release. These should only be used by Spark developers, as they +may have bugs and have not undergone the same level of testing as releases. Spark nightly packages +are available at: + +- Latest master build: <a href="http://people.apache.org/~pwendell/spark-nightly/spark-master-bin/latest";>http://people.apache.org/~pwendell/spark-nightly/spark-master-bin/latest</a> +- All nightly builds: <a href="http://people.apache.org/~pwendell/spark-nightly/";>http://people.apache.org/~pwendell/spark-nightly/</a> + +Spark also publishes SNAPSHOT releases of its Maven artifacts for both master and maintenance +branches on a nightly basis. To link to a SNAPSHOT you need to add the ASF snapshot +repository to your build. Note that SNAPSHOT artifacts are ephemeral and may change or +be removed. To use these you must add the ASF snapshot repository at +<a href="http://repository.apache.org/snapshots/";>http://repository.apache.org/snapshots/<a>. + +``` +groupId: org.apache.spark +artifactId: spark-core_2.10 +version: 1.5.0-SNAPSHOT +``` + +<a name="profiling"></a> +<h3>Profiling Spark Applications Using YourKit</h3> + +Here are instructions on profiling Spark applications using YourKit Java Profiler. + +<h4>On Spark EC2 images</h4> + +- After logging into the master node, download the YourKit Java Profiler for Linux from the +<a href="https://www.yourkit.com/download/index.jsp";>YourKit downloads page</a>. +This file is pretty big (~100 MB) and YourKit downloads site is somewhat slow, so you may +consider mirroring this file or including it on a custom AMI. +- Untar this file somewhere (in `/root` in our case): `tar xvjf yjp-12.0.5-linux.tar.bz2` +- Copy the expanded YourKit files to each node using copy-dir: `~/spark-ec2/copy-dir /root/yjp-12.0.5` +- Configure the Spark JVMs to use the YourKit profiling agent by editing `~/spark/conf/spark-env.sh` +and adding the lines +``` +SPARK_DAEMON_JAVA_OPTS+=" -agentpath:/root/yjp-12.0.5/bin/linux-x86-64/libyjpagent.so=sampling" +export SPARK_DAEMON_JAVA_OPTS +SPARK_JAVA_OPTS+=" -agentpath:/root/yjp-12.0.5/bin/linux-x86-64/libyjpagent.so=sampling" +export SPARK_JAVA_OPTS +``` +- Copy the updated configuration to each node: `~/spark-ec2/copy-dir ~/spark/conf/spark-env.sh` +- Restart your Spark cluster: `~/spark/bin/stop-all.sh` and `~/spark/bin/start-all.sh` +- By default, the YourKit profiler agents use ports 10001-10010. To connect the YourKit desktop +application to the remote profiler agents, you'll have to open these ports in the cluster's EC2 +security groups. To do this, sign into the AWS Management Console. Go to the EC2 section and +select `Security Groups` from the `Network & Security` section on the left side of the page. +Find the security groups corresponding to your cluster; if you launched a cluster named `test_cluster`, +then you will want to modify the settings for the `test_cluster-slaves` and `test_cluster-master` +security groups. For each group, select it from the list, click the `Inbound` tab, and create a +new `Custom TCP Rule` opening the port range `10001-10010`. Finally, click `Apply Rule Changes`. +Make sure to do this for both security groups. +Note: by default, `spark-ec2` re-uses security groups: if you stop this cluster and launch another +cluster with the same name, your security group settings will be re-used. +- Launch the YourKit profiler on your desktop. +- Select "Connect to remote application..." from the welcome screen and enter the the address of your Spark master or worker machine, e.g. `ec2--.compute-1.amazonaws.com` +- YourKit should now be connected to the remote profiling agent. It may take a few moments for profiling information to appear. + +Please see the full YourKit documentation for the full list of profiler agent +<a href="http://www.yourkit.com/docs/80/help/startup_options.jsp";>startup options</a>. + +<h4>In Spark unit tests</h4> + +When running Spark tests through SBT, add `javaOptions in Test += "-agentpath:/path/to/yjp"` +to `SparkBuild.scala` to launch the tests with the YourKit profiler agent enabled. +The platform-specific paths to the profiler agents are listed in the +<a href="http://www.yourkit.com/docs/80/help/agent.jsp";>YourKit documentation</a>. http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/documentation.md ---------------------------------------------------------------------- diff --git a/documentation.md b/documentation.md index 465f432..6234eac 100644 --- a/documentation.md +++ b/documentation.md @@ -174,13 +174,6 @@ Slides, videos and EC2-based exercises from each of these are available online: <li>The <a href="{{site.baseurl}}/examples.html">Spark examples page</a> shows the basic API in Scala, Java and Python.</li> </ul> -<h3>Wiki</h3> - -<ul><li> -The <a href="https://cwiki.apache.org/confluence/display/SPARK/Wiki+Homepage";>Spark wiki</a> contains -information for developers, such as architecture documents and how to <a href="{{site.baseurl}}/contributing.html">">contribute</a> to Spark. -</li></ul> - <h3>Research Papers</h3> <p> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/downloads.md ---------------------------------------------------------------------- diff --git a/downloads.md b/downloads.md index 17e1c7c..ddf9ba6 100644 --- a/downloads.md +++ b/downloads.md @@ -69,5 +69,5 @@ Once you've downloaded Spark, you can find instructions for installing and build <ul id="sparkReleaseNotes"></ul> ### Nightly Packages and Artifacts -For developers, Spark maintains nightly builds and SNAPSHOT artifacts. More information is available on the [Spark developer Wiki](https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-NightlyBuilds). +For developers, Spark maintains nightly builds and SNAPSHOT artifacts. More information is available on the [the Developer Tools page](/developer-tools.html#nightly-builds). http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/release-process.md ---------------------------------------------------------------------- diff --git a/release-process.md b/release-process.md new file mode 100644 index 0000000..0de2adc --- /dev/null +++ b/release-process.md @@ -0,0 +1,263 @@ +--- +layout: global +title: Release Process +type: "page singular" +navigation: + weight: 5 + show: true +--- + +<h2>Preparing Spark Releases</h2> + +<h3>Background</h3> + +The release manager role in Spark means you are responsible for a few different things: + +1. Preparing for release candidates: + 1. cutting a release branch + 1. informing the community of timing + 1. working with component leads to clean up JIRA + 1. making code changes in that branch with necessary version updates +1. Running the voting process for a release: + 1. creating release candidates using automated tooling + 1. calling votes and triaging issues +1. Finalizing and posting a release: + 1. updating the Spark website + 1. writing release notes + 1. announcing the release + +<h2>Preparing Spark for Release</h2> + +The main step towards preparing a release is to create a release branch. This is done via +standard Git branching mechanism and should be announced to the community once the branch is +created. It is also good to set up Jenkins jobs for the release branch once it is cut to +ensure tests are passing (consult Josh Rosen and Shane Knapp for help with this). + +Next, ensure that all Spark versions are correct in the code base on the release branch (see +<a href="https://github.com/apache/spark/commit/01d233e4aede65ffa39b9d2322196d4b64186526";>this example commit</a>). +You should grep through the codebase to find all instances of the version string. Some known +places to change are: + +- **SparkContext**. Search for VERSION (only for branch 1.x) +- **Maven build**. Ensure that the version in all the `pom.xml` files is `<SPARK-VERSION>-SNAPSHOT` +(e.g. `1.1.1-SNAPSHOT`). This will be changed to `<SPARK-VERSION>` (e.g. 1.1.1) automatically by +Maven when cutting the release. Note that there are a few exceptions that should just use +`<SPARK-VERSION>`, like `extras/java8-tests/pom.xml`. These modules are not published as artifacts. +- **Spark REPLs**. Look for the Spark ASCII art in `SparkILoopInit.scala` for the Scala shell +and in `shell.py` for the Python REPL. +- **Docs**. Search for VERSION in `docs/_config.yml` + +Finally, update `CHANGES.txt` with this script in the Spark repository. `CHANGES.txt` captures +all the patches that have made it into this release candidate since the last release. + +``` +$ export SPARK_HOME=<your Spark home> +$ cd spark +# Update release versions +$ vim dev/create-release/generate-changelist.py +$ dev/create-release/generate-changelist.py +``` + +This produces a `CHANGES.txt.new` that should be a superset of the existing `CHANGES.txt`. +Replace the old `CHANGES.txt` with the new one (see +<a href="https://github.com/apache/spark/commit/131c62672a39a6f71f6834e9aad54b587237f13c";>this example commit</a>). + +<h3>Cutting a Release Candidate</h3> + +If this is not the first RC, then make sure that the JIRA issues that have been solved since the +last RC are marked as `FIXED` in this release version. + +- A possible protocol for this is to mark such issues as `FIXED` in next maintenance release. +For example if you are cutting RC for 1.0.2, mark such issues as `FIXED` in 1.0.3. +- When cutting new RC, find all the issues that are marked as `FIXED` for next maintenance +release, and change them to the current release. +- Verify from `git log` whether they are actually making it in the new RC or not. + +The process of cutting a release candidate has been automated via the AMPLab Jenkins. There are +Jenkins jobs that can tag a release candidate and create various packages based on that candidate. +The recommended process is to ask the previous release manager to walk you through the Jenkins jobs. + +<h3>Call a Vote on the Release Candidate</h3> + +The release voting takes place on the Apache Spark developers list (the PMC is voting). +Look at past voting threads to see how this proceeds. The email should follow +<a href="http://mail-archives.apache.org/mod_mbox/spark-dev/201407.mbox/%3ccabpqxss7cf+yauuxck0jnush4207hcp4dkwn3bwfsvdnd86...@mail.gmail.com%3e";>this format</a>. + +- Make a shortened link to the full list of JIRAs using <a href="http://s.apache.org/";>http://s.apache.org/</a> +- If possible, attach a draft of the release notes with the email +- Make sure the voting closing time is in UTC format. Use this script to generate it +- Make sure the email is in text format and the links are correct + +Once the vote is done, you should also send out a summary email with the totals, with a subject +that looks something like `[RESULT] [VOTE]...`. + +<h3>Finalize the Release</h3> + +**Be Careful!** + +**THIS STEP IS IRREVERSIBLE so make sure you selected the correct staging repository. Once you +move the artifacts into the release folder, they cannot be removed.** + +After the vote passes, find the staging repository and click Release and confirm. To upload the +binaries, you have to first upload them to the dev directory in the Apache Distribution repo, +and then move the binaries from dev directory to release directory. This "moving" is the only + way you can add stuff to the actual release directory. + +``` +# Checkout the Spark directory in Apache distribution SVN "dev" repo +$ svn co https://dist.apache.org/repos/dist/dev/spark/ + +# Make directory for this RC in the above directory +mkdir spark-1.1.1-rc2 + +# Download the voted binaries and add them to the directory +$ scp andrewo...@people.apache.org:~/public_html/spark-1.1.1-rc2/* spark-1.1.1-rc2 + +# NOTE: Remove any binaries you donât want to publish +# E.g. never push MapR and *without-hive artifacts to apache +$ rm spark-1.1.1-rc2/*mapr* +$ rm spark-1.1.1-rc2/*without-hive* +$ svn add spark-1.1.1-rc2 +$ svn commit -m "Add spark-1.1.1-rc2" --username "andrewor14" + +# Move the sub-directory in "dev" to the +# corresponding directory in "release" +$ export SVN_EDITOR=vim +$ svn mv https://dist.apache.org/repos/dist/dev/spark/spark-1.1.1-rc2 https://dist.apache.org/repos/dist/release/spark/spark-1.1.1 +``` + +Verify that the resources are present in <a href="http://www.apache.org/dist/spark/";>http://www.apache.org/dist/spark/</a>. +It may take a while for them to be visible. This will be mirrored throughout the Apache network. +There are a few remaining steps. + +<h4>Remove Old Releases from Mirror Network</h4> + +Spark always keeps two releases in the mirror network: the most recent release on the current and +previous branches. To delete older versions simply use svn rm. The `downloads.js` file in the +website `js/` directory must also be updated to reflect the changes. For instance, the two +releases should be 1.1.1 and 1.0.2, but not 1.1.1 and 1.1.0. + +``` +$ svn rm https://dist.apache.org/repos/dist/release/spark/spark-1.1.0 +``` + +<h4>Update the Spark Apache Repository</h4> + +Check out the tagged commit for the release candidate that passed and apply the correct version tag. + +``` +$ git checkout v1.1.1-rc2 # the RC that passed +$ git tag v1.1.1 +$ git push apache v1.1.1 + +# Verify that the tag has been applied correctly +# If so, remove the old tag +$ git push apache :v1.1.1-rc2 +$ git tag -d v1.1.1-rc2 +``` + +Next, update remaining version numbers in the release branch. If you are doing a patch release, +see the similar commit made after the previous release in that branch. For example, for branch 1.0, +see <a href="https://github.com/apache/spark/commit/2a5514f7dcb9765b60cb772b97038cbbd1b58983";>this example commit</a>. + +In general, the rules are as follows: + +- `grep` through the repository to find such occurrences +- References to the version just released. Upgrade them to next release version. If it is not a +documentation related version (e.g. inside `spark/docs/` or inside `spark/python/epydoc.conf`), +add `-SNAPSHOT` to the end. +- References to the next version. Ensure these already have `-SNAPSHOT`. + +<!-- +<h4>Update the EC2 Scripts</h4> + +Upload the binary packages to the S3 bucket s3n://spark-related-packages (ask pwendell to do this). Then, change the init scripts in mesos/spark-ec2 repository to pull new binaries (see this example commit). +For Spark 1.1+, update branch v4+ +For Spark 1.1, update branch v3+ +For Spark 1.0, update branch v3+ +For Spark 0.9, update branch v2+ +You can audit the ec2 set-up by launching a cluster and running this audit script. Make sure you create cluster with default instance type (m1.xlarge). +--> + +<h4>Update the Spark Website</h4> + +The website repository is located at +<a href="https://github.com/apache/spark-website";>https://github.com/apache/spark-website</a>. +Ensure the docs were generated with the PRODUCTION=1 environment variable and with Java 7. + +``` +# Build the latest docs +$ git checkout v1.1.1 +$ cd docs +$ JAVA_HOME=$JAVA_7_HOME PRODUCTION=1 jekyll build + +# Copy the new documentation to apache +$ git clone https://github.com/apache/spark-website +... +$ cp -R _site spark-website/site/docs/1.1.1 + +# Update the "latest" link +$ cd spark/site/docs +$ rm latest +$ ln -s 1.1.1 latest +``` + +Next, update the rest of the Spark website. See how the previous releases are documented. +In particular, have a look at the changes to the `*.md` files in this commit (all the HTML +file changes are generated by `jekyll`). + +``` +$ git add 1.1.1 +$ git commit -m "Add docs for Spark 1.1.1" +``` + +Then, create the release notes. The contributors list can be automatically generated through +<a href="https://github.com/apache/spark/blob/branch-1.1/dev/create-release/generate-contributors.py";>this script</a>. +It accepts the tag that corresponds to the current release and another tag that +corresponds to the previous (not including maintenance release). For instance, if you are +releasing Spark 1.2.0, set the current tag to v1.2.0-rc2 and the previous tag to v1.1.0. +Once you have generated the initial contributors list, it is highly likely that there will be +warnings about author names not being properly translated. To fix this, run +<a href="https://github.com/apache/spark/blob/branch-1.1/dev/create-release/translate-contributors.py";>this other script</a>, +which fetches potential replacements from Github and JIRA. For instance: + +``` +$ cd release-spark/dev/create-release +# Set RELEASE_TAG and PREVIOUS_RELEASE_TAG +$ vim generate-contributors.py +# Generate initial contributors list, likely with warnings +$ ./generate-contributors.py +# Set JIRA_USERNAME, JIRA_PASSWORD, and GITHUB_API_TOKEN +$ vim release-spark/dev/translate-contributors.py +# Translate names generated in the previous step, reading from known_translations if necessary +$ ./translate-contributors.py +``` + +Additionally, if you wish to give more specific credit for developers of larger patches, you may +use the the following commands to identify large patches. Extra care must be taken to make sure +commits from previous releases are not counted since git cannot easily associate commits that +were back ported into different branches. + +``` +# Determine PR numbers closed only in the new release +$ git log v1.1.1 | grep "Closes #" | cut -d " " -f 5,6 | grep Closes | sort > closed_1.1.1 +$ git log v1.1.0 | grep "Closes #" | cut -d " " -f 5,6 | grep Closes | sort > closed_1.1.0 +$ diff --new-line-format="" --unchanged-line-format="" closed_1.1.1 closed_1.1.0 > diff.txt + +# Grep expression with all new patches +$ EXPR=$(cat diff.txt | awk '{ print "\\("$1" "$2" \\)"; }' | tr "\n" "|" | sed -e "s/|/\\\|/g" | sed "s/\\\|$//") + +# Contributor list +$ git shortlog v1.1.1 --grep "$EXPR" > contrib.txt + +# Large patch list (300+ lines) +$ git log v1.1.1 --grep "$expr" --shortstat --oneline | grep -B 1 -e "[3-9][0-9][0-9] insert" -e "[1-9][1-9][1-9][1-9] insert" | grep SPARK > large-patches.txt +``` + +Then, update the downloads page, and then the main page with a news item. + +<h4>Create an Announcement</h4> + +Once everything is working (website docs, website changes) create an announcement on the website +and then send an e-mail to the mailing list. Enjoy an adult beverage of your choice, and +congratulations on making a Spark release. http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/committers.html ---------------------------------------------------------------------- diff --git a/site/committers.html b/site/committers.html index bad4414..5069db0 100644 --- a/site/committers.html +++ b/site/committers.html @@ -108,24 +108,32 @@ <ul class="dropdown-menu"> <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="/examples.html">Examples</a></li> <li class="dropdown"> - <a href="/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="/community.html">Mailing Lists & Resources</a></li> <li><a href="/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="/community.html#events">Events and Meetups</a></li> - <li><a href="/community.html#history">Project History</a></li> <li><a href="/powered-by.html">Powered By</a></li> <li><a href="/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/community.html ---------------------------------------------------------------------- diff --git a/site/community.html b/site/community.html index 1a5e80f..a83ae1b 100644 --- a/site/community.html +++ b/site/community.html @@ -108,24 +108,32 @@ <ul class="dropdown-menu"> <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="/examples.html">Examples</a></li> <li class="dropdown"> - <a href="/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="/community.html">Mailing Lists & Resources</a></li> <li><a href="/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="/community.html#events">Events and Meetups</a></li> - <li><a href="/community.html#history">Project History</a></li> <li><a href="/powered-by.html">Powered By</a></li> <li><a href="/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/contributing.html ---------------------------------------------------------------------- diff --git a/site/contributing.html b/site/contributing.html index 72b5292..be01615 100644 --- a/site/contributing.html +++ b/site/contributing.html @@ -108,24 +108,32 @@ <ul class="dropdown-menu"> <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="/examples.html">Examples</a></li> <li class="dropdown"> - <a href="/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="/community.html">Mailing Lists & Resources</a></li> <li><a href="/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="/community.html#events">Events and Meetups</a></li> - <li><a href="/community.html#history">Project History</a></li> <li><a href="/powered-by.html">Powered By</a></li> <li><a href="/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/developer-tools.html ---------------------------------------------------------------------- diff --git a/site/developer-tools.html b/site/developer-tools.html new file mode 100644 index 0000000..0de49c6 --- /dev/null +++ b/site/developer-tools.html @@ -0,0 +1,494 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title> + Useful Developer Tools | Apache Spark + + </title> + + + + + + <!-- Bootstrap core CSS --> + <link href="/css/cerulean.min.css" rel="stylesheet"> + <link href="/css/custom.css" rel="stylesheet"> + + <!-- Code highlighter CSS --> + <link href="/css/pygments-default.css" rel="stylesheet"> + + <script type="text/javascript"> + <!-- Google Analytics initialization --> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-32518208-2']); + _gaq.push(['_trackPageview']); + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + + <!-- Adds slight delay to links to allow async reporting --> + function trackOutboundLink(link, category, action) { + try { + _gaq.push(['_trackEvent', category , action]); + } catch(err){} + + setTimeout(function() { + document.location.href = link.href; + }, 100); + } + </script> + + <!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js";></script> + <script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js";></script> + <![endif]--> +</head> + +<body> + +<script src="https://code.jquery.com/jquery.js";></script> +<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.3/js/bootstrap.min.js";></script> +<script src="/js/lang-tabs.js"></script> +<script src="/js/downloads.js"></script> + +<div class="container" style="max-width: 1200px;"> + +<div class="masthead"> + + <p class="lead"> + <a href="/"> + <img src="/images/spark-logo-trademark.png" + style="height:100px; width:auto; vertical-align: bottom; margin-top: 20px;"></a><span class="tagline"> + Lightning-fast cluster computing + </span> + </p> + +</div> + +<nav class="navbar navbar-default" role="navigation"> + <!-- Brand and toggle get grouped for better mobile display --> + <div class="navbar-header"> + <button type="button" class="navbar-toggle" data-toggle="collapse" + data-target="#navbar-collapse-1"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + </div> + + <!-- Collect the nav links, forms, and other content for toggling --> + <div class="collapse navbar-collapse" id="navbar-collapse-1"> + <ul class="nav navbar-nav"> + <li><a href="/downloads.html">Download</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Libraries <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/sql/">SQL and DataFrames</a></li> + <li><a href="/streaming/">Spark Streaming</a></li> + <li><a href="/mllib/">MLlib (machine learning)</a></li> + <li><a href="/graphx/">GraphX (graph)</a></li> + <li class="divider"></li> + <li><a href="/third-party-projects.html">Third-Party Projects</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Documentation <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> + <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> + </ul> + </li> + <li><a href="/examples.html">Examples</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Community <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/community.html">Mailing Lists & Resources</a></li> + <li><a href="/contributing.html">Contributing to Spark</a></li> + <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> + <li><a href="/powered-by.html">Powered By</a></li> + <li><a href="/committers.html">Project Committers</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> + </ul> + <ul class="nav navbar-nav navbar-right"> + <li class="dropdown"> + <a href="http://www.apache.org/"; class="dropdown-toggle" data-toggle="dropdown"> + Apache Software Foundation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/";>Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </li> + </ul> + </div> + <!-- /.navbar-collapse --> +</nav> + + +<div class="row"> + <div class="col-md-3 col-md-push-9"> + <div class="news" style="margin-bottom: 20px;"> + <h5>Latest News</h5> + <ul class="list-unstyled"> + + <li><a href="/news/spark-wins-cloudsort-100tb-benchmark.html">Spark wins CloudSort Benchmark as the most efficient engine</a> + <span class="small">(Nov 15, 2016)</span></li> + + <li><a href="/news/spark-2-0-2-released.html">Spark 2.0.2 released</a> + <span class="small">(Nov 14, 2016)</span></li> + + <li><a href="/news/spark-1-6-3-released.html">Spark 1.6.3 released</a> + <span class="small">(Nov 07, 2016)</span></li> + + <li><a href="/news/spark-2-0-1-released.html">Spark 2.0.1 released</a> + <span class="small">(Oct 03, 2016)</span></li> + + </ul> + <p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p> + </div> + <div class="hidden-xs hidden-sm"> + <a href="/downloads.html" class="btn btn-success btn-lg btn-block" style="margin-bottom: 30px;"> + Download Spark + </a> + <p style="font-size: 16px; font-weight: 500; color: #555;"> + Built-in Libraries: + </p> + <ul class="list-none"> + <li><a href="/sql/">SQL and DataFrames</a></li> + <li><a href="/streaming/">Spark Streaming</a></li> + <li><a href="/mllib/">MLlib (machine learning)</a></li> + <li><a href="/graphx/">GraphX (graph)</a></li> + </ul> + <a href="/third-party-projects.html">Third-Party Projects</a> + </div> + </div> + + <div class="col-md-9 col-md-pull-3"> + <h2>Useful Developer Tools</h2> + +<h3>Reducing Build Times</h3> + +<p>Spark’s default build strategy is to assemble a jar including all of its dependencies. This can +be cumbersome when doing iterative development. When developing locally, it is possible to create +an assembly jar including all of Spark’s dependencies and then re-package only Spark itself +when making changes.</p> + +<pre><code>$ build/sbt clean package +$ ./bin/spark-shell +$ export SPARK_PREPEND_CLASSES=true +$ ./bin/spark-shell # Now it's using compiled classes +# ... do some local development ... # +$ build/sbt compile +# ... do some local development ... # +$ build/sbt compile +$ unset SPARK_PREPEND_CLASSES +$ ./bin/spark-shell + +# You can also use ~ to let sbt do incremental builds on file changes without running a new sbt session every time +$ build/sbt ~compile +</code></pre> + +<h3>Checking Out Pull Requests</h3> + +<p>Git provides a mechanism for fetching remote pull requests into your own local repository. +This is useful when reviewing code or testing patches locally. If you haven’t yet cloned the +Spark Git repository, use the following command:</p> + +<pre><code>$ git clone https://github.com/apache/spark.git +$ cd spark +</code></pre> + +<p>To enable this feature you’ll need to configure the git remote repository to fetch pull request +data. Do this by modifying the <code>.git/config</code> file inside of your Spark directory. The remote may +not be named “origin” if you’ve named it something else:</p> + +<pre><code>[remote "origin"] + url = g...@github.com:apache/spark.git + fetch = +refs/heads/*:refs/remotes/origin/* + fetch = +refs/pull/*/head:refs/remotes/origin/pr/* # Add this line +</code></pre> + +<p>Once you’ve done this you can fetch remote pull requests</p> + +<pre><code># Fetch remote pull requests +$ git fetch origin +# Checkout a remote pull request +$ git checkout origin/pr/112 +# Create a local branch from a remote pull request +$ git checkout origin/pr/112 -b new-branch +</code></pre> + +<h3>Generating Dependency Graphs</h3> + +<pre><code>$ # sbt +$ build/sbt dependency-tree + +$ # Maven +$ build/mvn -DskipTests install +$ build/mvn dependency:tree +</code></pre> + +<p><a name="individual-tests"></a></p> +<h3>Running Build Targets For Individual Projects</h3> + +<pre><code>$ # sbt +$ build/sbt package +$ # Maven +$ build/mvn package -DskipTests -pl assembly +</code></pre> + +<h3>ScalaTest Issues</h3> + +<p>If the following error occurs when running ScalaTest</p> + +<pre><code>An internal error occurred during: "Launching XYZSuite.scala". +java.lang.NullPointerException +</code></pre> +<p>It is due to an incorrect Scala library in the classpath. To fix it:</p> + +<ul> + <li>Right click on project</li> + <li>Select <code>Build Path | Configure Build Path</code></li> + <li><code>Add Library | Scala Library</code></li> + <li>Remove <code>scala-library-2.10.4.jar - lib_managed\jars</code></li> +</ul> + +<p>In the event of “Could not find resource path for Web UI: org/apache/spark/ui/static”, +it’s due to a classpath issue (some classes were probably not compiled). To fix this, it +sufficient to run a test from the command line:</p> + +<pre><code>build/sbt "test-only org.apache.spark.rdd.SortingSuite" +</code></pre> + +<h3>Running Different Test Permutations on Jenkins</h3> + +<p>When running tests for a pull request on Jenkins, you can add special phrases to the title of +your pull request to change testing behavior. This includes:</p> + +<ul> + <li><code>[test-maven]</code> - signals to test the pull request using maven</li> + <li><code>[test-hadoop1.0]</code> - signals to test using Spark’s Hadoop 1.0 profile (other options include +Hadoop 2.0, 2.2, and 2.3)</li> +</ul> + +<h3>Organizing Imports</h3> + +<p>You can use a <a href="https://plugins.jetbrains.com/plugin/7350";>IntelliJ Imports Organizer</a> +from Aaron Davidson to help you organize the imports in +your code. It can be configured to match the import ordering from the style guide.</p> + +<h3>IDE Setup</h3> + +<h4>IntelliJ</h4> + +<p>While many of the Spark developers use SBT or Maven on the command line, the most common IDE we +use is IntelliJ IDEA. You can get the community edition for free (Apache committers can get +free IntelliJ Ultimate Edition licenses) and install the JetBrains Scala plugin from <code>Preferences > Plugins</code>.</p> + +<p>To create a Spark project for IntelliJ:</p> + +<ul> + <li>Download IntelliJ and install the +<a href="https://confluence.jetbrains.com/display/SCA/Scala+Plugin+for+IntelliJ+IDEA";>Scala plug-in for IntelliJ</a>.</li> + <li>Go to <code>File -> Import Project</code>, locate the spark source directory, and select “Maven Project”.</li> + <li>In the Import wizard, it’s fine to leave settings at their default. However it is usually useful +to enable “Import Maven projects automatically”, since changes to the project structure will +automatically update the IntelliJ project.</li> + <li>As documented in <a href="http://spark.apache.org/docs/latest/building-spark.html";>Building Spark</a>, +some build configurations require specific profiles to be +enabled. The same profiles that are enabled with <code>-P[profile name]</code> above may be enabled on the +Profiles screen in the Import wizard. For example, if developing for Hadoop 2.4 with YARN support, +enable profiles yarn and hadoop-2.4. These selections can be changed later by accessing the +“Maven Projects” tool window from the View menu, and expanding the Profiles section.</li> +</ul> + +<p>Other tips:</p> + +<ul> + <li>“Rebuild Project” can fail the first time the project is compiled, because generate source files +are not automatically generated. Try clicking the “Generate Sources and Update Folders For All +Projects” button in the “Maven Projects” tool window to manually generate these sources.</li> + <li>Some of the modules have pluggable source directories based on Maven profiles (i.e. to support +both Scala 2.11 and 2.10 or to allow cross building against different versions of Hive). In some +cases IntelliJ’s does not correctly detect use of the maven-build-plugin to add source directories. +In these cases, you may need to add source locations explicitly to compile the entire project. If +so, open the “Project Settings” and select “Modules”. Based on your selected Maven profiles, you +may need to add source folders to the following modules: + <ul> + <li>spark-hive: add v0.13.1/src/main/scala</li> + <li>spark-streaming-flume-sink: add target\scala-2.10\src_managed\main\compiled_avro</li> + </ul> + </li> + <li>Compilation may fail with an error like “scalac: bad option: +-P:/home/jakub/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar”. +If so, go to Preferences > Build, Execution, Deployment > Scala Compiler and clear the “Additional +compiler options” field. It will work then although the option will come back when the project +reimports. If you try to build any of the projects using quasiquotes (eg., sql) then you will +need to make that jar a compiler plugin (just below “Additional compiler options”). +Otherwise you will see errors like: + <pre><code>/Users/irashid/github/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala +Error:(147, 9) value q is not a member of StringContext + Note: implicit class Evaluate2 is not applicable here because it comes after the application point and it lacks an explicit result type + q""" + ^ +</code></pre> + </li> +</ul> + +<h4>Eclipse</h4> + +<p>Eclipse can be used to develop and test Spark. The following configuration is known to work:</p> + +<ul> + <li>Eclipse Juno</li> + <li><a href="http://scala-ide.org/";>Scala IDE 4.0</a></li> + <li>Scala Test</li> +</ul> + +<p>The easiest way is to download the Scala IDE bundle from the Scala IDE download page. It comes +pre-installed with ScalaTest. Alternatively, use the Scala IDE update site or Eclipse Marketplace.</p> + +<p>SBT can create Eclipse <code>.project</code> and <code>.classpath</code> files. To create these files for each Spark sub +project, use this command:</p> + +<pre><code>sbt/sbt eclipse +</code></pre> + +<p>To import a specific project, e.g. spark-core, select <code>File | Import | Existing Projects</code> into +Workspace. Do not select “Copy projects into workspace”.</p> + +<p>If you want to develop on Scala 2.10 you need to configure a Scala installation for the +exact Scala version thatâs used to compile Spark. + Since Scala IDE bundles the latest versions (2.10.5 and 2.11.8 at this point), you need to add one +in <code>Eclipse Preferences -> Scala -> Installations</code> by pointing to the <code>lib/</code> directory of your +Scala 2.10.5 distribution. Once this is done, select all Spark projects and right-click, +choose <code>Scala -> Set Scala Installation</code> and point to the 2.10.5 installation. +This should clear all errors about invalid cross-compiled libraries. A clean build should succeed now.</p> + +<p>ScalaTest can execute unit tests by right clicking a source file and selecting <code>Run As | Scala Test</code>.</p> + +<p>If Java memory errors occur, it might be necessary to increase the settings in <code>eclipse.ini</code> +in the Eclipse install directory. Increase the following setting as needed:</p> + +<pre><code>--launcher.XXMaxPermSize +256M +</code></pre> + +<p><a name="nightly-builds"></a></p> +<h3>Nightly Builds</h3> + +<p>Packages are built regularly off of Spark’s master branch and release branches. These provide +Spark developers access to the bleeding-edge of Spark master or the most recent fixes not yet +incorporated into a maintenance release. These should only be used by Spark developers, as they +may have bugs and have not undergone the same level of testing as releases. Spark nightly packages +are available at:</p> + +<ul> + <li>Latest master build: <a href="http://people.apache.org/~pwendell/spark-nightly/spark-master-bin/latest";>http://people.apache.org/~pwendell/spark-nightly/spark-master-bin/latest</a></li> + <li>All nightly builds: <a href="http://people.apache.org/~pwendell/spark-nightly/";>http://people.apache.org/~pwendell/spark-nightly/</a></li> +</ul> + +<p>Spark also publishes SNAPSHOT releases of its Maven artifacts for both master and maintenance +branches on a nightly basis. To link to a SNAPSHOT you need to add the ASF snapshot +repository to your build. Note that SNAPSHOT artifacts are ephemeral and may change or +be removed. To use these you must add the ASF snapshot repository at +<a href="http://repository.apache.org/snapshots/";>http://repository.apache.org/snapshots/<a>.</a></a></p> + +<pre><code>groupId: org.apache.spark +artifactId: spark-core_2.10 +version: 1.5.0-SNAPSHOT +</code></pre> + +<p><a name="profiling"></a></p> +<h3>Profiling Spark Applications Using YourKit</h3> + +<p>Here are instructions on profiling Spark applications using YourKit Java Profiler.</p> + +<h4>On Spark EC2 images</h4> + +<ul> + <li>After logging into the master node, download the YourKit Java Profiler for Linux from the +<a href="https://www.yourkit.com/download/index.jsp";>YourKit downloads page</a>. +This file is pretty big (~100 MB) and YourKit downloads site is somewhat slow, so you may +consider mirroring this file or including it on a custom AMI.</li> + <li>Untar this file somewhere (in <code>/root</code> in our case): <code>tar xvjf yjp-12.0.5-linux.tar.bz2</code></li> + <li>Copy the expanded YourKit files to each node using copy-dir: <code>~/spark-ec2/copy-dir /root/yjp-12.0.5</code></li> + <li>Configure the Spark JVMs to use the YourKit profiling agent by editing <code>~/spark/conf/spark-env.sh</code> +and adding the lines + <pre><code>SPARK_DAEMON_JAVA_OPTS+=" -agentpath:/root/yjp-12.0.5/bin/linux-x86-64/libyjpagent.so=sampling" +export SPARK_DAEMON_JAVA_OPTS +SPARK_JAVA_OPTS+=" -agentpath:/root/yjp-12.0.5/bin/linux-x86-64/libyjpagent.so=sampling" +export SPARK_JAVA_OPTS +</code></pre> + </li> + <li>Copy the updated configuration to each node: <code>~/spark-ec2/copy-dir ~/spark/conf/spark-env.sh</code></li> + <li>Restart your Spark cluster: <code>~/spark/bin/stop-all.sh</code> and <code>~/spark/bin/start-all.sh</code></li> + <li>By default, the YourKit profiler agents use ports 10001-10010. To connect the YourKit desktop +application to the remote profiler agents, you’ll have to open these ports in the cluster’s EC2 +security groups. To do this, sign into the AWS Management Console. Go to the EC2 section and +select <code>Security Groups</code> from the <code>Network & Security</code> section on the left side of the page. +Find the security groups corresponding to your cluster; if you launched a cluster named <code>test_cluster</code>, +then you will want to modify the settings for the <code>test_cluster-slaves</code> and <code>test_cluster-master</code> +security groups. For each group, select it from the list, click the <code>Inbound</code> tab, and create a +new <code>Custom TCP Rule</code> opening the port range <code>10001-10010</code>. Finally, click <code>Apply Rule Changes</code>. +Make sure to do this for both security groups. +Note: by default, <code>spark-ec2</code> re-uses security groups: if you stop this cluster and launch another +cluster with the same name, your security group settings will be re-used.</li> + <li>Launch the YourKit profiler on your desktop.</li> + <li>Select “Connect to remote application…” from the welcome screen and enter the the address of your Spark master or worker machine, e.g. <code>ec2--.compute-1.amazonaws.com</code></li> + <li>YourKit should now be connected to the remote profiling agent. It may take a few moments for profiling information to appear.</li> +</ul> + +<p>Please see the full YourKit documentation for the full list of profiler agent +<a href="http://www.yourkit.com/docs/80/help/startup_options.jsp";>startup options</a>.</p> + +<h4>In Spark unit tests</h4> + +<p>When running Spark tests through SBT, add <code>javaOptions in Test += "-agentpath:/path/to/yjp"</code> +to <code>SparkBuild.scala</code> to launch the tests with the YourKit profiler agent enabled.<br /> +The platform-specific paths to the profiler agents are listed in the +<a href="http://www.yourkit.com/docs/80/help/agent.jsp";>YourKit documentation</a>.</p> + + </div> +</div> + + + +<footer class="small"> + <hr> + Apache Spark, Spark, Apache, and the Spark logo are <a href="/trademarks.html">trademarks</a> of + <a href="http://www.apache.org";>The Apache Software Foundation</a>. +</footer> + +</div> + +</body> +</html> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/documentation.html ---------------------------------------------------------------------- diff --git a/site/documentation.html b/site/documentation.html index b8a8b40..8d911da 100644 --- a/site/documentation.html +++ b/site/documentation.html @@ -108,24 +108,32 @@ <ul class="dropdown-menu"> <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="/examples.html">Examples</a></li> <li class="dropdown"> - <a href="/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="/community.html">Mailing Lists & Resources</a></li> <li><a href="/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="/community.html#events">Events and Meetups</a></li> - <li><a href="/community.html#history">Project History</a></li> <li><a href="/powered-by.html">Powered By</a></li> <li><a href="/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> @@ -350,13 +358,6 @@ Slides, videos and EC2-based exercises from each of these are available online: <li>The <a href="/examples.html">Spark examples page</a> shows the basic API in Scala, Java and Python.</li> </ul> -<h3>Wiki</h3> - -<ul><li> -The <a href="https://cwiki.apache.org/confluence/display/SPARK/Wiki+Homepage";>Spark wiki</a> contains -information for developers, such as architecture documents and how to <a href="/contributing.html">">contribute</a> to Spark. -</li></ul> - <h3>Research Papers</h3> <p> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/downloads.html ---------------------------------------------------------------------- diff --git a/site/downloads.html b/site/downloads.html index 6bb636d..aa88bba 100644 --- a/site/downloads.html +++ b/site/downloads.html @@ -108,24 +108,32 @@ <ul class="dropdown-menu"> <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="/examples.html">Examples</a></li> <li class="dropdown"> - <a href="/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="/community.html">Mailing Lists & Resources</a></li> <li><a href="/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="/community.html#events">Events and Meetups</a></li> - <li><a href="/community.html#history">Project History</a></li> <li><a href="/powered-by.html">Powered By</a></li> <li><a href="/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> @@ -256,7 +264,7 @@ git clone git://github.com/apache/spark.git -b branch-2.0 <ul id="sparkReleaseNotes"></ul> <h3 id="nightly-packages-and-artifacts">Nightly Packages and Artifacts</h3> -<p>For developers, Spark maintains nightly builds and SNAPSHOT artifacts. More information is available on the <a href="https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-NightlyBuilds";>Spark developer Wiki</a>.</p> +<p>For developers, Spark maintains nightly builds and SNAPSHOT artifacts. More information is available on the <a href="/developer-tools.html#nightly-builds">the Developer Tools page</a>.</p> </div> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/examples.html ---------------------------------------------------------------------- diff --git a/site/examples.html b/site/examples.html index 877e5fa..18ece51 100644 --- a/site/examples.html +++ b/site/examples.html @@ -108,24 +108,32 @@ <ul class="dropdown-menu"> <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="/examples.html">Examples</a></li> <li class="dropdown"> - <a href="/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="/community.html">Mailing Lists & Resources</a></li> <li><a href="/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="/community.html#events">Events and Meetups</a></li> - <li><a href="/community.html#history">Project History</a></li> <li><a href="/powered-by.html">Powered By</a></li> <li><a href="/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/faq.html ---------------------------------------------------------------------- diff --git a/site/faq.html b/site/faq.html index 49ec5e3..cc8e0b9 100644 --- a/site/faq.html +++ b/site/faq.html @@ -108,24 +108,32 @@ <ul class="dropdown-menu"> <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="/examples.html">Examples</a></li> <li class="dropdown"> - <a href="/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="/community.html">Mailing Lists & Resources</a></li> <li><a href="/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="/community.html#events">Events and Meetups</a></li> - <li><a href="/community.html#history">Project History</a></li> <li><a href="/powered-by.html">Powered By</a></li> <li><a href="/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/graphx/index.html ---------------------------------------------------------------------- diff --git a/site/graphx/index.html b/site/graphx/index.html index 3a1d235..d3251bf 100644 --- a/site/graphx/index.html +++ b/site/graphx/index.html @@ -111,24 +111,32 @@ <ul class="dropdown-menu"> <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="/examples.html">Examples</a></li> <li class="dropdown"> - <a href="/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="/community.html">Mailing Lists & Resources</a></li> <li><a href="/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="/community.html#events">Events and Meetups</a></li> - <li><a href="/community.html#history">Project History</a></li> <li><a href="/powered-by.html">Powered By</a></li> <li><a href="/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/index.html ---------------------------------------------------------------------- diff --git a/site/index.html b/site/index.html index b35ec09..0ddfb81 100644 --- a/site/index.html +++ b/site/index.html @@ -110,24 +110,32 @@ <ul class="dropdown-menu"> <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="/examples.html">Examples</a></li> <li class="dropdown"> - <a href="/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="/community.html">Mailing Lists & Resources</a></li> <li><a href="/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="/community.html#events">Events and Meetups</a></li> - <li><a href="/community.html#history">Project History</a></li> <li><a href="/powered-by.html">Powered By</a></li> <li><a href="/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/mailing-lists.html ---------------------------------------------------------------------- diff --git a/site/mailing-lists.html b/site/mailing-lists.html index b0383fe..c113cdd 100644 --- a/site/mailing-lists.html +++ b/site/mailing-lists.html @@ -111,24 +111,32 @@ <ul class="dropdown-menu"> <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="/examples.html">Examples</a></li> <li class="dropdown"> - <a href="/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="/community.html">Mailing Lists & Resources</a></li> <li><a href="/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="/community.html#events">Events and Meetups</a></li> - <li><a href="/community.html#history">Project History</a></li> <li><a href="/powered-by.html">Powered By</a></li> <li><a href="/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/mllib/index.html ---------------------------------------------------------------------- diff --git a/site/mllib/index.html b/site/mllib/index.html index 3eb63eb..11808fb 100644 --- a/site/mllib/index.html +++ b/site/mllib/index.html @@ -111,24 +111,32 @@ <ul class="dropdown-menu"> <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="/examples.html">Examples</a></li> <li class="dropdown"> - <a href="/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="/community.html">Mailing Lists & Resources</a></li> <li><a href="/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="/community.html#events">Events and Meetups</a></li> - <li><a href="/community.html#history">Project History</a></li> <li><a href="/powered-by.html">Powered By</a></li> <li><a href="/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> http://git-wip-us.apache.org/repos/asf/spark-website/blob/cf21826b/site/news/amp-camp-2013-registration-ope.html ---------------------------------------------------------------------- diff --git a/site/news/amp-camp-2013-registration-ope.html b/site/news/amp-camp-2013-registration-ope.html index 067d74a..b9d1aba 100644 --- a/site/news/amp-camp-2013-registration-ope.html +++ b/site/news/amp-camp-2013-registration-ope.html @@ -108,24 +108,32 @@ <ul class="dropdown-menu"> <li><a href="/docs/latest/">Latest Release (Spark 2.0.2)</a></li> <li><a href="/documentation.html">Older Versions and Other Resources</a></li> + <li><a href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li><a href="/examples.html">Examples</a></li> <li class="dropdown"> - <a href="/community.html" class="dropdown-toggle" data-toggle="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> Community <b class="caret"></b> </a> <ul class="dropdown-menu"> - <li><a href="/community.html#mailing-lists">Mailing Lists</a></li> + <li><a href="/community.html">Mailing Lists & Resources</a></li> <li><a href="/contributing.html">Contributing to Spark</a></li> <li><a href="https://issues.apache.org/jira/browse/SPARK";>Issue Tracker</a></li> - <li><a href="/community.html#events">Events and Meetups</a></li> - <li><a href="/community.html#history">Project History</a></li> <li><a href="/powered-by.html">Powered By</a></li> <li><a href="/committers.html">Project Committers</a></li> </ul> </li> - <li><a href="/faq.html">FAQ</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown"> + Developers <b class="caret"></b> + </a> + <ul class="dropdown-menu"> + <li><a href="/developer-tools.html">Useful Developer Tools</a></li> + <li><a href="/versioning-policy.html">Versioning Policy</a></li> + <li><a href="/release-process.html">Release Process</a></li> + </ul> + </li> </ul> <ul class="nav navbar-nav navbar-right"> <li class="dropdown"> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
